mapbox / geojson-merge

Merge multiple GeoJSON files into one FeatureCollection.
ISC License
233 stars 33 forks source link

"Invalid string length" when merge a large files #33

Open anphanngocvn opened 5 years ago

anphanngocvn commented 5 years ago

I have a few .json files (some of them about 750Mb) need to be merged into one .geofile. However when I run this command to merge them. geojson-merge folder/*.geojson > combined.geojson. I have following errors:

`/geojson-merge/geojson-merge:17 process.stdout.write(JSON.stringify(merge(argv._.map(function(n) { ^

RangeError: Invalid string length at JSON.stringify () at Object. (/geojson-merge/geojson-merge:17:29) at Module._compile (internal/modules/cjs/loader.js:701:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10) at Module.load (internal/modules/cjs/loader.js:600:32) at tryModuleLoad (internal/modules/cjs/loader.js:539:12) at Function.Module._load (internal/modules/cjs/loader.js:531:3) at Function.Module.runMain (internal/modules/cjs/loader.js:754:12) at startup (internal/bootstrap/node.js:283:19) at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)`

How do you tackle this?

gorbypark commented 5 years ago

Hi, I ran into the same problem a few weeks ago. The issue is that you can only add so many arguments to a command (maybe 1000 on bash/linux? I can't remember,). The only solution to this is to do a batch operation (for example, if you have 10,000 files to merge, merge 500 at a time until you have 20 files, and then merge those 20 together). The solution I took was to write a quick node script that iterated over every file in a folder and merges that way. There is not really a max file type restriction in node when iterating over a folder (although you could risk running out of ram!).

qrcf commented 5 years ago

I have the same issue on the entire Kantō region (roads only) of Japan. Its just merging two files. not hundreds. Apparently the file itself is too big at 769.3mb

warrenca commented 5 years ago

@anphanngocvn @qrcf which version of node are you using? i encountered the problem in v6.10, tried in v11 and it's all working fine. I still trying to find out now which node version was it really fixed.

ghost commented 3 years ago

Might want to remove the 2 from https://github.com/mapbox/geojson-merge/blob/fccdc8d5a53f13f535ee6c538a6edf2d98c57e76/geojson-merge#L19, indentation seems to inflate file sizes.

jackcjp commented 2 years ago

Might want to remove the 2 from

https://github.com/mapbox/geojson-merge/blob/fccdc8d5a53f13f535ee6c538a6edf2d98c57e76/geojson-merge#L19

, indentation seems to inflate file sizes.

Change 2 to 0 works for me! thank you @cagriozkurt