Maximum call stack size exceeded

[tafflin]

Hi, I've been trying to create a stops-dissolved-buffer geojson out of 395000 stops. The config file looks like this: { "agencies": [ { "agency_key": "allstops1", "path": "/home/tafflin/Documents/gtfs-to-geojson/stops1.zip", "exclude": [ "stop_times", "trips", "shapes" ] } ], "bufferSizeMeters": 800, "coordinatePrecision": 5, "outputType": "agency", "outputFormat": "stops-dissolved", "zipOutput": false }

The gtfs contains only a stops.txt file with necessary fields and agency.txt file with 1 test agency. When I try to create a multi polygon geojson with stops buffer I get an error of "Maximum call stack size exceeded". What is the maximum number of stops that can be processed and how can I increase this number if possible? Smaller files with ~12000 stops have no issues and are generated in ~10 sec.

[brendannee]

That is a lot of stops - I have not figured out what the max number is.

The --max-old-space-size= flag won't be used correctly when running gtfs-to-geojson as a globally installed npm package.

Instead, you could clone the repo, then from the root directory

node --max-old-space-size=30000 ./bin/gtfs-to-geojson to see if that helps.

Can you share your stops.txt file with me and I'll test it out to see where things can be optimized?

[tafflin]

Look like this command before starting the utility solved the issue with "out of memory" problem I had before I encountered "maximum call stack": export NODE_OPTIONS="--max-old-space-size=30000"

Here is a zip with 347k stops. Get the same error "maximum call stack" with it. stops.zip

[brendannee]

Thanks for the notes.

I think you may need to process a smaller number of stops as non-dissolved stops geojson, then merge the resulting geojson files together and create a dissolved buffer using a tool like QGIS.

[tafflin]

Should I leave the issue open?

By any chance do you know if QGIS can be used as CLI and which command should be used to create a dissolved stops geojson? I intended to use gtfs-to-geojson in a data-processing pipeline

[brendannee]

I pushed an updated version that solves this issue:

https://github.com/BlinkTagInc/gtfs-to-geojson/releases/tag/3.2.0

It replaces the polygon-clipping library which was throwing that error with the @turf/union library.

The @turf/union function only takes two shapes at a time to compute a union, so it ends up being pretty slow with 400,000 stops, but it doesn't run out of memory.

Check it out and let me know what you think.

[brendannee]

If you are interested, you can see the code that manages the dissolve of buffered stops here: https://github.com/BlinkTagInc/gtfs-to-geojson/blob/master/lib/geojson-utils.js#L98-L110

[tafflin]

Did you have an opportunity to check if my attached file works with the new version? I still get the same error