search

mapbox / csv2geojson

magically convert csv files to geojson files
http://mapbox.github.io/csv2geojson/
MIT License
364 stars 82 forks source link

Cannot create a string longer than 0x1fffffe8 characters #79

Open chrisbitmead opened 1 year ago

chrisbitmead commented 1 year ago

So... it would seem that the code tries to slurp the whole file into memory in one go, and convert it to one big string. That's not a good idea....


Error: Cannot create a string longer than 0x1fffffe8 characters
    at Buffer.toString (node:buffer:784:17)
    at convert (C:\Users\chris\GITW\nsldist\client\node_modules\csv2geojson\csv2geojson:23:34)
    at ConcatStream.<anonymous> (C:\Users\chris\GITW\nsldist\client\node_modules\concat-stream\index.js:36:43)
    at ConcatStream.emit (node:events:402:35)
    at finishMaybe (C:\Users\chris\GITW\nsldist\client\node_modules\readable-stream\lib\_stream_writable.js:475:14)
    at endWritable (C:\Users\chris\GITW\nsldist\client\node_modules\readable-stream\lib\_stream_writable.js:485:3)
    at ConcatStream.Writable.end (C:\Users\chris\GITW\nsldist\client\node_modules\readable-stream\lib\_stream_writable.js:455:41)
    at ReadStream.onend (node:internal/streams/readable:693:10)
    at Object.onceWrapper (node:events:509:28)
    at ReadStream.emit (node:events:390:28) {
  code: 'ERR_STRING_TOO_LONG'
}```
tracek commented 1 year ago

I run into this one too. Plus, a similar one, the same root cause:

/home/tracek/.npm-packages/lib/node_modules/csv2geojson/csv2geojson:46
            console.log(JSON.stringify(data, null, 2));
                             ^

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at /home/tracek/.npm-packages/lib/node_modules/csv2geojson/csv2geojson:46:30
    at Object.csv2geojson (/home/tracek/.npm-packages/lib/node_modules/csv2geojson/index.js:190:5)
    at convert (/home/tracek/.npm-packages/lib/node_modules/csv2geojson/csv2geojson:40:17)
    at ConcatStream.<anonymous> (/home/tracek/.npm-packages/lib/node_modules/csv2geojson/node_modules/concat-stream/index.js:36:43)
    at ConcatStream.emit (node:events:525:35)
    at finishMaybe (/home/tracek/.npm-packages/lib/node_modules/csv2geojson/node_modules/readable-stream/lib/_stream_writable.js:475:14)
    at endWritable (/home/tracek/.npm-packages/lib/node_modules/csv2geojson/node_modules/readable-stream/lib/_stream_writable.js:485:3)
    at ConcatStream.Writable.end (/home/tracek/.npm-packages/lib/node_modules/csv2geojson/node_modules/readable-stream/lib/_stream_writable.js:455:41)
    at ReadStream.onend (node:internal/streams/readable:693:10)