search

topojson / topojson-server

Convert GeoJSON to TopoJSON.
ISC License
117 stars 32 forks source link

geo2topo fails on large files #15

Closed JoeGermuska closed 5 years ago

JoeGermuska commented 5 years ago

I'm trying to produce a topojson file of all ZCTAs in the US. I converted the Census TIGER data to geojson (1.3 GB), and then when I try to convert it to topojson, I get the error below.

$ geo2topo zcta5.geo.json >  zcta5.topo.json 
buffer.js:644
    return this.utf8Slice(0, this.length);
                ^

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.toString (buffer.js:644:17)
    at JSON.parse (<anonymous>)
    at ReadStream.<anonymous> (/Users/germuska/.nvm/versions/node/v10.4.1/lib/node_modules/topojson/node_modules/topojson-server/bin/geo2topo:107:46)
    at ReadStream.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1081:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

As my next step, I extracted roughly half the data (based on ZCTA ids beginning with 0-4). This produced a 609M geojson file. When executing geo2topo on that file, I got a different error: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory from the v8 JsonParser

I tried again, splitting into 5 groups, and got the same error on a geojson file of around 300MB.

When I split into 10 geojson files, they all make it through correctly, but now I realize that I don't know a simple way to put them all back together again!

Can anything be done to produce topojson from large geodata files?

mbostock commented 5 years ago

I recommend using newline-delimited JSON input so that you can stream one feature at a time. That will work for larger inputs, though of course you can still run out of memory. You can also pass arguments to Node to give it more memory.