monav-preprocessor: bad_alloc

[GoogleCodeExporter]

I downloaded the osm europe map from [1] and ran monav-preprocessor 
-s=europe.ini -t=2 -v -di -dro -dre -da -dc on it. (europe.ini attached)

The result is usually:
$ monav-preprocessor -s=europe.ini -t=2 -v -di -dro -dre -da -dc
[15:30:40][0] OSM Importer: access list: "motorcar" 
[15:30:40][0] OSM Importer: access list: "motor_vehicle" 
[15:30:40][0] OSM Importer: access list: "vehicle" 
[15:30:40][0] OSM Importer: access list: "access" 
[15:30:40][0] OSM Importer: access list: "bicycle" 
[15:30:40][0] OSM Importer: access list: "vehicle" 
[15:30:40][0] OSM Importer: access list: "access" 
[15:30:40][0] OSM Importer:: loaded speed profile: ":/speed 
profiles/bicycle.spp" 
[15:46:03][0] OSM Importer: finished import pass 1: 923464 ms 
[15:50:11][0] OSM Importer: filtered node coordinates: 212826 ms 
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

In GDB it looks a bit different:
(gdb) run -s=europe.ini -t=2 -v -di -dro -dre -da -dc
Starting program: /usr/bin/monav-preprocessor -s=europe.ini -t=2 -v -di -dro 
-dre -da -dc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[16:04:38][0] OSM Importer: access list: "motorcar" 
[16:04:38][0] OSM Importer: access list: "motor_vehicle" 
[16:04:38][0] OSM Importer: access list: "vehicle" 
[16:04:38][0] OSM Importer: access list: "access" 
[16:04:38][0] OSM Importer: access list: "bicycle" 
[16:04:38][0] OSM Importer: access list: "vehicle" 
[16:04:38][0] OSM Importer: access list: "access" 
[16:04:38][0] OSM Importer:: loaded speed profile: ":/speed 
profiles/bicycle.spp" 
[16:26:39][0] OSM Importer: finished import pass 1: 1320622 ms 
Catchpoint 1 (exception thrown), 0x00000036286b7da0 in __cxa_throw () from 
/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.2/libstdc++.so.6

Backtrace attached.

This was with MoNav 0.3 on Gentoo/Linux.

[1] 
http://ftp5.gwdg.de/pub/misc/openstreetmap/download.geofabrik.de/europe.osm.pbf

Original issue reported on code.google.com by dschridde@gmail.com on 8 Feb 2012 at 3:39

Attachments:

• europe.ini
• monav-preprocessor.backtrace.log

[GoogleCodeExporter]

How much memory do you have available? Importing the whole of Europe can take 
up quite some memory. E.g. when I checked last year it was about 8GB.

Original comment by veaac.fd...@gmail.com on 8 Feb 2012 at 10:42

[GoogleCodeExporter]

The whole system has 6G with about 1G in use already. My observation was that 
there was still a few gig left while monav-preprocessor was running, but I 
would have to rerun to confirm. Are huge chunks (several gig) of memory being 
allocated at once? That might explain why I missed the memory usage rising 
during the run.

In any case it would probably be nice to inform the user that he is lacking RAM 
to complete the operation, instead of going down hard.

Original comment by dschridde@gmail.com on 9 Feb 2012 at 1:54

[GoogleCodeExporter]

There is one huge amount of memory being allocated ( to determine which nodes 
are part of which city ). There would be a way to do this more space efficient 
but about 2-3 times slower.

I should definitely catch the exception and report a more meaningful error 
message, though.

Original comment by veaac.fd...@gmail.com on 9 Feb 2012 at 8:34