bad_alloc issue taking all osm relations

[cayetanobv]

Hi @cvvergara ,

This is the bad_alloc issue with osm_relations we was reviewing yesterday in your branch https://github.com/cvvergara/osm2pgrouting/tree/test2

With this commit https://github.com/cvvergara/osm2pgrouting/commit/32c7a7bf4f2c309366c255b9eac33f696aba02d9 the bad_alloc does not raise because you don't take all the relations.

This happened in my dockerized environment (on a Ubuntu host). In the next days I'll try to reproduce this error with the same dataset but without Docker. Dataset: https://drive.google.com/open?id=1IyRLf0mjzFI5fx9y2WO5DboFjTw8X--k

Command line:

$ osm2pgrouting --file /tmp/output_data.osm --dbname routing --user routing_admin --password routing --port 5432 --host localhost --schema test_schema --clean --addnodes --tags --attributes

Shell output:

***************************************************
           COMMAND LINE CONFIGURATION             *
***************************************************
Filename = /tmp/output_data.osm
Configuration file = /usr/src/app/itinera/mapconfig/mapconfig.xml
host = localhost
port = 5432
dbname = routing
username = routing_admin
password = routing
schema= test_schema
prefix = 
suffix = 
Drop tables
Don't create indexes
Add OSM nodes
***************************************************
Testing database connection: routing
database connection successful: routing
Connecting to the database
connection success

Dropping tables...
TABLE: test_schema.ways dropped ... OK.
TABLE: test_schema.ways_vertices_pgr dropped ... OK.
TABLE: test_schema.pointsofinterest dropped ... OK.
TABLE: test_schema.configuration dropped ... OK.
TABLE: test_schema.osm_nodes dropped ... OK.
TABLE: test_schema.osm_ways dropped ... OK.
TABLE: test_schema.osm_relations dropped ... OK.

Creating tables...
TABLE: test_schema.ways_vertices_pgr created ... OK.
TABLE: test_schema.ways created ... OK.
TABLE: test_schema.pointsofinterest created ... OK.
TABLE: test_schema.configuration created ... OK.
TABLE: test_schema.osm_nodes created ... OK.
TABLE: test_schema.osm_ways created ... OK.
TABLE: test_schema.osm_relations created ... OK.
Opening configuration file: /usr/src/app/itinera/mapconfig/mapconfig.xml
    Parsing configuration

Exporting configuration ...
  - Done 
Counting lines ...
  - Done 
Opening data file: /tmp/output_data.osm total lines: 5150594
    Parsing data

Current osm_nodes:  1580000
Final osm_nodes:    1599211
Current osm_ways:   160000
Final osm_ways: 177210

Final osm_relations:    4470std::bad_alloc

Thanks,

[cayetanobv]

Hi @cvvergara ,

I compiled without Docker your branch test2 and the bad_alloc raises again. I realized that application never enters in this IF statement https://github.com/cvvergara/osm2pgrouting/blob/test2/src/osm_elements/OSMDocument.cpp#L125

if (m_relations.size() % m_chunk_size == 0) {
            wait_child();
            std::cout << "Current osm_relations:\t" << m_relations.size();
            osm_table_export(m_relations, "osm_relations");
            m_relPending = false;
        }

Is there maybe something wrong with this behaviour?

Thanks,

[cvvergara]

@cayetanobv I leave it to you as homework:

• if it never enters that part of the code where the data gets exported to the database then tell me how come you get relations in the database on the first place?

• if the only change is: that test2 has

  } else {
  m_rDocument.AddRelation(*last_relation);

https://github.com/cvvergara/osm2pgrouting/commit/7143d65c91356c8f33761b475d2a25ff84cc96f3#diff-9940549a437a5f80764f6b1d2461cdc9

I still can't replicate your error. bad::alloc is when you run out of memory

So I might have more memory than you

[cayetanobv]

Hi @cvvergara ,

Sorry for the delay in response... too much work in the last week.

I put some prints in include/database/Export2DB.h to see whats going on and a try/catch block to avoid finish insert data process (and explore database):

https://github.com/cayetanobv/osm2pgrouting/blob/cayetano/test2/include/database/Export2DB.h#L89

void export_osm (
       std::vector<T> &items,
       const std::string &table) const {
   auto osm_table = m_tables.get_table(table);
   std::vector<std::string> values(items.size(), "");
   try{
     size_t i(0);
     for (auto it = items.begin(); it != items.end(); ++it, ++i) {
         auto item = *it;
         values[i] = tab_separated(item.values(osm_table.columns(), true));
         if (table == "osm_relations"){
            std::cout << "\nNEW ROW XXXX\t" << values[i];
            std::cout << "\n-- XXXX\t" << i;
         }
     }
   } catch (const std::bad_alloc& e) {
      std::cout << "\nERROR: " << e.what() << '\n';
   }

   export_osm(values, osm_table);
}

Shell output is too long with this prints, so I attached this file with all output info: output.txt

Bad_alloc raises after relation with osm_id=73447 is inserted in the database (you can see that this is a very big osm relation).

Thanks!

[cvvergara]

Ok, Well, because its the fist time it really spits out relations, we will use the one that works for you. Maybe when using libosmium we can fix that kind of big relations issue. But not now.

[cayetanobv]

Ok @cvvergara , but... do we keep this try/catch block inside export_osm() function (removing innecesary prints) to allow the insertion to database of all possible relations until some error raises or we mantain your commit ?

[cayetanobv]

Work in progress for this issue in: https://github.com/cvvergara/osm2pgrouting/pull/13

[cayetanobv]

We can close this issue @cvvergara . This is fixed with this PR https://github.com/pgRouting/osm2pgrouting/pull/212

[hetongde]

Is this problem solved? I converted this problem, is not too much data, osm data 3g, osm2pgrouting version 2.3.3.
Converted data size can not exceed 2g? print:

Exporting configuration ...

• Done Counting lines ...

• Done Opening data file: usa1_ways.osm total lines: 77327590 Parsing data std::bad_alloc

  @cvvergara @cayetanobv

Thanks!

[cayetanobv]

Hi @hetongde , the problem is solved in the context of this issue, but if you want I could test your config with your data. I would need the following:

• osm2pgrouting shell command with all the flags you are using.
• I need either your dataset (with a link to Drive or Dropbox) or your BBOX (x_min, y_min, x_max, y_max).
• What are your hardware specifications (at least RAM)?

REMARK: I would say that everything is a matter of RAM memory in this version yet. In future versions this issue will be improved (cc: @cvvergara).

Thanks,

[hetongde]

Thanks verymuch @cayetanobv ,i using osmconvert convert us-northeast-latest.osm.pdf to osm,then using osmfilter extract highway from osm ，this osm file about 3.5g,then using osm2pgrouting to get topo. 1 . osm2pgrouting sell command is: osm2pgrouting --f usa1_ways.osm --conf "/usr/share/osm2pgrouting/mapconfig_for_cars.xml" --dbname usalarge --username osm --password 123456

2.RAM total used free shared buffers cached Mem: 16063392 4493504 11569888 332008 316988 2873620

[cayetanobv]

Could you pass me exact filter applied to osmfilter (copy and paste shell command) to get same data as you? Thanks.

[hetongde]

Ok， @cayetanobv There are two main steps：

1. osmconvert us-northeast-latest.osm.pbf --out-o5m -o=usa1.o5m
2. osmfilter usa1.o5m --keep="highway =motorway = motorway_link =motorway_junction =trunk = trunk_link =primary = primary_link =secondary =secondary_link =tertiaty = tertiaty_link = residential= living_street = service =unclassified = road" --drop-author --drop-version --drop="access=no" -o=usa1_ways.osm On my computer, I split the data to 2g size is possible. Osmconver usa1_ways.osm -b=-77.7866461,38.9296804,-71,47.4949548 --complete-ways -o=usa112.osm I found that the segmentation must be - complete-ways, otherwise there will be a lot of long lines, and the line is chaotic.

[cvvergara]

@hetongde I dont think that osm2pgRouting can handle 2G in memory, I think you need to break the file into sections and load it by parts. (unless you are using a very big server) So, break your bounding box into 4, a little overlapping, bounding boxes and load the 4 files. one at a time.

[hetongde]

@cvvergara Ok,After I split the file, it succeeded. Thanks!

[cayetanobv]

@hetongde great!