importing osm ends in killed

[gplv2]

On Ubuntu 13.04 - raring:

Importing benelux subset ends in killed, using up all memory (2gig machine). fyi: Nominatim import of pbf file processed fine with postgresql settings.

top memory right before kill:

2537 postgres  20   0 2138m 1.8g  316 R  81.8 91.8   1:03.82 osm2pgrouting     

postgres@osm:~$ osm2pgrouting -file /usr/local/src/planet-benelux-130726.osm -conf /usr/local/src/mapconfig.xml -dbname routing -user postgres -clean 

host=127.0.0.1 user=postgres dbname=routing port=5432 connection success Trying to load config file /usr/local/src/mapconfig.xml Trying to parse config Trying to load data Trying to parse data Killed

How much memory would one actually need to process this?

root@osm:~# dpkg --list | grep postg | grep -v java
ii  postgis                                  1.5.3-2ubuntu1                      amd64        Geographic objects support for PostgreSQL -- common files
ii  postgresql                               9.1+140                             all          object-relational SQL database (supported version)
ii  postgresql-9.1                           9.1.9-1ubuntu1                      amd64        object-relational SQL database, version 9.1 server
ii  postgresql-9.1-pgrouting                 2.0.0-rc1-ppa17                     amd64        Routing functionality support for PostgreSQL/PostGIS
ii  postgresql-9.1-postgis                   1.5.3-2ubuntu1                      amd64        Geographic objects support for PostgreSQL 9.1
ii  postgresql-client-9.1                    9.1.9-1ubuntu1                      amd64        front-end programs for PostgreSQL 9.1
ii  postgresql-client-common                 140                                 all          manager for multiple PostgreSQL client versions
ii  postgresql-common                        140                                 all          PostgreSQL database-cluster manager
ii  postgresql-contrib                       9.1+140                             all          additional facilities for PostgreSQL (supported version)
ii  postgresql-contrib-9.1                   9.1.9-1ubuntu1                      amd64        additional facilities for PostgreSQL
ii  postgresql-server-dev-9.1                9.1.9-1ubuntu1                      amd64        development files for PostgreSQL 9.1 server-side programming

postgresql error

2013-07-28 16:21:07 UTC LOG:  could not receive data from client: Connection reset by peer
2013-07-28 16:21:07 UTC LOG:  unexpected EOF on client connection

strace says:

...
read(4, "9586\" version=\"1\"/>\n\t<node id=\"6"..., 8192) = 8192
read(4, "n=\"1\"/>\n\t<node id=\"680886887\" la"..., 8192) = 8192
read(4, "ode id=\"680887060\" lat=\"51.92231"..., 8192) = 8192
read(4, "7345\" lat=\"51.918674\" lon=\"4.612"..., 8192) = 8192
read(4, "\"51.9152755\" lon=\"4.580735\" vers"..., 8192) = 8192
read(4, " lon=\"4.612746\" version=\"2\"/>\n\t<"..., 8192) = 8192
read(4, ".5918497\" version=\"1\"/>\n\t<node i"..., 8192) = 8192
read(4, "5935244\" version=\"1\"/>\n\t<node id"..., 8192) = 8192
brk(0x81dad000)                         = 0x81dad000
read(4, ".5969857\" version=\"1\"/>\n\t<node i"..., 8192) = 8192
read(4, "6095411\" version=\"1\"/>\n\t<node id"..., 8192) = 8192
read(4, "rsion=\"1\"/>\n\t<node id=\"680889170"..., 8192) = 8192
+++ killed by SIGKILL +++
Killed

What can I do to help locate, reproduce and fix this? Any information you need? Things to try out ? Love to assist.

[woodbri]

This sounds like osm2pgrouting was killed by the OOM (out of memory) kernel process because the system ran out of memory and swap space. Try increasing the amount of swap space you have allocated, like 2X or 4X you physical memory.

[gplv2]

Hey tx. The file to import is almost 9gig, that's only Benelux, been looking at the code , by the looks of it, it will try the load the file into memory in 1 go ... I can only image what a planet import would need... I'll try it though.

[gplv2]

I was low on swap (linode VM) so I created a 9 gig swap file to cover for the source size. hope it's enough. I've been thinking ... Would it work to just split up the OSM source files. I guess without the -clean parameter this - in theory- would have to work if I account for the XML structure, no?

I'm actually at the next stage it seems right now, tx for the suggestion, as it turned out. Only thing I didn't try was a 'dmesg' , that tells me the memory is exhausted at one point.

[dkastl]

This is a limitation of osm2pgrouting that has been there from the beginning. With additional swap space you can work around this limitation, though osm2pgrouting is definitely not doing well with large areas so far.

Instead of keeping everything in memory it would make sense to write to disk when running out of memory. Other tools that process OSM data do so as well. If someone would implement this, lot of users might appreciate.

[gplv2]

Tried with plenty of swapspace, unfortunately the machine became unresponsive, except for a ping reply. Too bad, Might be not such a bad idea to look at how Routino does it, really fast way of indexing and splitting the jobs up in parts. Wish I could help but I'm a bad C-coder since age of 12 :)

[vlasvlasvlas]

here i do have the same problem..

[cvvergara]

Version 2.1, will also have the same problem, it will not be able to load in one call a continent, or a large coutry, but it will allow to make "incremental" additions to the database, so, for example to load a continent, maybe do it by loading each country into the same database.

[vlasvlasvlas]

loading by country works great.

[cvvergara]

Good to hear, it will remain as a feature request, because if the "incremental" works I need to figure out how to do "incremental" on one big file

[amcirillo]

Is there any update on this issue? I was just trying to import North America on a machine that has 64GB of memory and ran out. My osm file is 168GB in size and it sure would be nice to not have to try to do this piecemeal.

[cvvergara]

No update, but loading Canada, United States of Mexico and and USA separately or maybe by state/province

[cayetanobv]

@amcirillo You can read more about how to solve your problem here:

• https://github.com/pgRouting/osm2pgrouting/issues/221
• https://github.com/pgRouting/osm2pgrouting/issues/222

[cayetanobv]

MORE: @amcirillo You have 2 strategies:

• Incremental additions, cutting your area in several files.
• Increment your Swap space while your are importing if you don't have more RAM. It's slow but it works fine.

FUTURE: https://github.com/pgRouting/osm2pgrouting/issues/245