search

pgRouting / osm2pgrouting

Import tool for OpenStreetMap data to pgRouting database
https://pgrouting.org
GNU General Public License v2.0
289 stars 111 forks source link

Memory leak. #173

Open tocmorf opened 7 years ago

tocmorf commented 7 years ago

This not app. this single very big memory leak.

woodbri commented 7 years ago

Can you provide more information? What version? How can we reproduce the problem? What makes you think this is a memory leak and not that it just uses lots of memory?

tocmorf commented 7 years ago

more information - only after week end. But in file OSMDocumentParserCallback.cpp method OSMDocumentParserCallback::EndElement. add remark at line m_rDocument.AddNode(last_node); and start parsing big file (for example: http://download.geofabrik.de/europe/ukraine-latest.osm.bz2). after finish parsing nodes block (before start processing ways and relations) - memory usage very big, but nodes container is empty. "delete last_node" does not release memory. Maybe the problem is missing classes destructors Element, Node, Tag.

dkastl commented 7 years ago

@tocmorf osm2pgrouting depends on enough memory to handle larger OSM extracts. So this is eventually not a leak but the way osm2pgrouting processes data.

@cvvergara from which version is it possible to split the OSM data into smaller areas and process them one by one into the same database?

woodbri commented 7 years ago

On 5/5/2017 8:49 PM, Daniel Kastl wrote:

@tocmorf https://github.com/tocmorf osm2pgrouting depends on enough memory to handle larger OSM extracts. So this is eventually not a leak but the way osm2pgrouting processes data.

@cvvergara https://github.com/cvvergara from which version is it possible to split the OSM data into smaller areas and process them one by one into the same database?

Right, like I said there is a difference between a memory leak and just high memory usage. osm2pgrouting needs enough memory to load every thing into memory, it can not page data to disk. It was designed this way rightly or wrongly, but it is the design.

If you are interested in refactoring the code so this is not the case, then talk to @cvvergara about generating pull request.

This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

cvvergara commented 7 years ago

@dkastl From version 2.1it can do incremental additions to the database.