search

topojson / topojson-server

Convert GeoJSON to TopoJSON.
ISC License
117 stars 32 forks source link

Solution for ERR_STRING_TOO_LONG? #18

Closed hyperknot closed 4 years ago

hyperknot commented 4 years ago

I'm trying to run geo2topo on the GADM dataset: 6 layers, 7.3 GB in total. My problem is that even with raising the memory by node --max-old-space-size=20000, I cannot convert multiple layers of that dataset.

The process always exits with the following message:

buffer.js:763
    return this.utf8Slice(0, this.length);
                ^

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.toString (buffer.js:763:17)
    at JSON.parse (<anonymous>)
    at ReadStream.<anonymous> (.../country-levels/node_modules/topojson-server/bin/geo2topo:107:46)
    at ReadStream.emit (events.js:323:22)
    at endReadableNT (_stream_readable.js:1204:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_STRING_TOO_LONG'

The server has 64 GB of RAM, so it should be able to handle this in memory.

mbostock commented 4 years ago

It’s a limitation of Node, sorry. Try using newline-delimited input. Also, mind asking for help on Stack Overflow rather than posting issues here?