search

pgRouting / osm2pgrouting

Import tool for OpenStreetMap data to pgRouting database
https://pgrouting.org
GNU General Public License v2.0
288 stars 112 forks source link

osm2pgrouting takes forever #221

Open walpino opened 6 years ago

walpino commented 6 years ago

Expected behavior and actual behavior it should do the import in x hours

Steps to reproduce the problem Connecting to the database connection success

Dropping tables... TABLE: hackways droped ... OK. TABLE: hackways_vertices_pgr droped ... OK. TABLE: hackpointsofinterest droped ... OK. TABLE: configuration droped ... OK. TABLE: osm_nodes droped ... OK. TABLE: osm_ways droped ... OK. TABLE: osm_relations droped ... OK.

Creating tables... TABLE: hackways_vertices_pgr created ... OK. TABLE: hackways created ... OK. TABLE: hackpointsofinterest created ... OK. TABLE: configuration created ... OK. TABLE: osm_nodes created ... OK. TABLE: osm_ways created ... OK. TABLE: osm_relations created ... OK. Opening configuration file: /usr/share/osm2pgrouting/mapconfig.xml Parsing configuration

Exporting configuration ...

Done Counting lines ... Done Opening data file: planet_8.4572,47.3346_8.5867,47.4007.osm total lines: 1542029 Parsing data Current osm_nodes: 400000 Final osm_nodes: 401143 Current osm_ways: 0 Final osm_ways: 65171

it stucks there for hours and hours and I dont see why and what it s doing the osm file is just a small one

Specifications like the version of pgRouting/PostGIS and PostgreSQL as well as Operating System cnetos SELECT version(); version PostgreSQL 9.6.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16), 64-bit (1 row)

SELECT postgis_full_version(); postgis_full_version POSTGIS="2.4.3 r16312" PGSQL="96" GEOS="3.6.2-CAPI-1.10.2 4d2925d6" PROJ="Rel. 4.9.3, 15 August 2016" GDAL="GDAL 1.11.4, released 2016/01/25" LIBXML="2.9.1" LIBJSON="0.11" RASTER (1 row)

SELECT pgr_version(); pgr_version (2.5.2,v2.5.2,60585f1f7,master,1.53.0) (1 row)

Use the commands:

SELECT version(); SELECT postgis_full_version(); SELECT pgr_version(); any idea where it is stucked? or what takes so long? the machine has 10 cpus and 50 gb of ram

cayetanobv commented 6 years ago

Hi @walpino . What is the osm2pgrouting version you are using? Could you paste the output of this command?

osm2pgrouting --version

Could you share your OSM dataset (or bounding box you are using to export OSM dataset)?

I'm seeing file total lines (aprox 1.5M) and it is very little for 50GB RAM. I'm processing right now without problems >4M lines with only 16GB RAM.

Thanks,

Cayetano

znmeb commented 6 years ago

I'm testing with the state of Oregon - the whole thing takes about 20 GB of RAM and the .osm file is 3.3 GB. I have a smaller region that I'm really interested in - the TriMet (local transit system) service area. That's only a 1.4 GB .osm file and only needs 8.6 GB of RAM to load into the database.

cayetanobv commented 6 years ago

Hi @znmeb ,

If you didn't removed metadata tags (not necessary for PgRouting) you can reduce a lot the size of your Oregon OSM file following these steps (you can get around half size of original file; 3.3GB to 1.7GB aprox):

Thanks for using osm2pgrouting!

Cayetano

znmeb commented 6 years ago

@cayetanobv Thanks! That helped a bunch - I was doing the bounding box filtering with osmosis but osmconvert is faster and the resulting .osm output is only 678 megabytes. In my environment, the peak resident set size in osm2pgrouting is about 6.1 times the input size, which means I'll be a bit over 4 GB now.