search

marook / osm-read

an openstreetmap XML and PBF data parser for node.js and the browser
GNU Lesser General Public License v3.0
107 stars 25 forks source link

Allocation failed - process out of memory #35

Closed exebetche closed 1 year ago

exebetche commented 7 years ago

Hi,

I'd like to use this module to gather data from the pbf file extract for europe (http://download.geofabrik.de/europe-latest.osm.pbf, 20Go), but the process quickly run out of memory and fail. I tried to increase the memory available for nodejs to 8Go (--max-old-space-size=8192) but it fails anyway. Is there a way to parse large files with this module?

Here is the output:

<--- Last few GCs --->

  264433 ms: Scavenge 8195.2 (8358.6) -> 8195.2 (8358.6) MB, 8.8 / 0 ms (+ 1.8 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  270716 ms: Mark-sweep 8195.2 (8358.6) -> 8189.8 (8354.6) MB, 6283.4 / 0 ms (+ 2.5 ms in 2 steps since start of marking, biggest step 1.8 ms) [last resort gc].
  276850 ms: Mark-sweep 8189.8 (8354.6) -> 8190.2 (8358.6) MB, 6134.3 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x12b629ab4629 <JS Object>
    1: node [/osmread-js/pbfTest.js:~24] [pc=0x17d45f2c517c] (this=0x2360fd08bf11 <an Object with map 0xbe1c1a7d6d9>,node=0x179a95efcaf9 <an Object with map 0xbe1c1a6a8b1>)
    2: visitPrimitiveGroup(aka visitPrimitiveGroup) [/osmread-js/lib/pbfParser.js:~158] [pc=0x17d45f2f64d7] (this=0x12b629a041b9 <undefined>,pg=0x2355483166a1 <JS Object>,o...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abandon (core dumped)

nodejs version: 4.2.6 cmd: nodejs --max-old-space-size=8192 ./pbfTest.js

Thanks.

marook commented 1 year ago

I'm not sure what you are doing in the node, way or relation callbacks. These functions right now only support blocked processing. #56 is about adding support for asynchronous processing which might also fix this issue.