search

hackoregon / data-science-pet-containers

Pet Containers for Data Scientists
MIT License
11 stars 1 forks source link

osm2pgrouting crashes on Oregon data import #24

Closed znmeb closed 6 years ago

znmeb commented 6 years ago

This is upstream - options are to delete the application or to install newer version from source. Will try install route first (pun intended)

znmeb commented 6 years ago

I think the crash was operator error - I had the wrong download format for osm2pgrouting.

However, the build from source is working and it's the same version (2.3.3) that ships with EnterpriseDB on Windows, so I'm keeping it.

So this is now officially an enhancement.

P.S.: The script to test it crashes in Windows Docker - looks like 2 GB isn't enough for something this big.

znmeb commented 6 years ago

Definitely out of memory - running on a 32 GB workstation and the osm2pgrouting process has acquired an RSS of 16GB. This is a non-starter writ large.

Seems to top out at 21 GB - for all of Oregon. Truncating to TriMet service area should make this viable.

znmeb commented 6 years ago

Truncating to TriMet bounding box is a big win - it now peaks at 8.6 GB. That's still out of range for many laptops but this may be the fastest way to get this dataset loaded.

Rule of thumb appears to be: .osm file size * 6.1 = RAM required

znmeb commented 6 years ago

Peak RAM down to 4.2 GB - it should run (native) in an 8 GB laptop now! Thanks to https://github.com/pgRouting/osm2pgrouting/issues/221#issuecomment-372061947

znmeb commented 6 years ago

Closing - released