search

mbloch / mapshaper

Tools for editing Shapefile, GeoJSON, TopoJSON and CSV files
http://mapshaper.org
Other
3.78k stars 535 forks source link

Parsing of some shape files triggers "Maximum call stack size exceeded" error #518

Closed jaltin closed 2 years ago

jaltin commented 2 years ago

Hi,

We are seeing this error firing for some shape files when we try to parse them.

The layers causing the error are all coming from one specific source (an external source that we have no knowledge of where it originates, so we can't follow this up at the source).

When we parse the shape files with other software it works ok (we tried QGIS, DotSpatial and https://maps.equatorstudios.com), but when we use mapshaper it fires the following error:

RangeError: Maximum call stack size exceeded at new ShapeRecord (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:20963:39) at readShapeAtOffset (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21336:19) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21265:15) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18) at readNextShape (C:\home\site\wwwroot\node_modules\mapshaper\mapshaper.js:21275:18)

We are trying to do the parsing via applyCommand in an Azure function and that's where we are getting the error.

If we run it on a local machine with a simple node app (still using applyCommand), it parses ok.

If we try to parse the shapefile on https://mapshaper.org I also get the error.

This to me indicates that there is some max threshold on stack depth that Azure functions environment and the web browser hits, but not happening on my local machine.

Could it be a bug in Mapshaper or is the shape file faulty? Since other software can process the file, I hope there is a possible fix for this in mapshaper so that it can also process the layer.

I have attached a sample shapefile that triggers the error.

max_callstack_error.zip

Let me know if I can help further.

ThomasG77 commented 2 years ago

Quite similar to #519 where I've made an answer too. Magically solved with the following and like before do not have a good explanation (annoyed...)

ogr2ogr -t_srs "EPSG:3021" max_callstack_error_check_proj.shp max_callstack_error.shp
zip max_callstack_error_check_proj_zip.zip max_callstack_error_check_proj.*

Test in mapshaper GUI and "it works"

Differences when using ogrinfo seem minor e.g mainly extend

ogrinfo -so max_callstack_error.shp max_callstack_error > max_callstack_error_meta.txt
ogrinfo -so max_callstack_error_check_proj.shp max_callstack_error_check_proj > max_callstack_error_check_proj_meta.txt
diff max_callstack_error_meta.txt max_callstack_error_check_proj_meta.txt

Only visible difference is the extent in this case (see diff output below)

1c1
< INFO: Open of `max_callstack_error.shp'
---
> INFO: Open of `max_callstack_error_check_proj.shp'
4c4
< Layer name: max_callstack_error
---
> Layer name: max_callstack_error_check_proj
6c6
<   DBF_DATE_LAST_UPDATE=2017-05-05
---
>   DBF_DATE_LAST_UPDATE=2021-12-10
9c9
< Extent: (1505085.237055, 7025374.852674) - (1670483.282797, 7089626.686511)
---
> Extent: (1512603.330043, 7028295.390575) - (1662965.189809, 7086706.148609)

Result files below:

mbloch commented 2 years ago

I'll investigate this. Note that ogrinfo (GDAL version 3.3.1) also displays an error when I run ogrinfo max_callstack_error.shp:

INFO: Open of `max_callstack_error.shp'
      using driver `ESRI Shapefile' successful.
1: max_callstack_error (Line String)
jaltin commented 2 years ago

Thanks @mbloch and @ThomasG77 for looking into this issue!

Yes it might be that the source of these shape files are somehow producing faulty files and that it can't be fixed.

However, It is curious that the other softwares I mentioned manages to handle it though. And also the fact that it ran ok on local machine but not in mapshaper website or in our Azure functions app.

Not much more I can contribute with technically at this moment, but let me know if I can help in any way.

ThomasG77 commented 2 years ago

@mbloch Same answer https://github.com/mbloch/mapshaper/issues/519#issuecomment-991250482 as concerning the supposed error from GDAL output

mbloch commented 2 years ago

As I explained in my comment to issue #519, I updated mapshaper so that these files can be imported.

Brief explanation -- the files don't quite conform to the Shapefile spec, but are still readable. The "maximum stack size exceeded error" was caused by a bug in mapshaper.

Thanks for reporting this and helping to improve mapshaper!

jaltin commented 2 years ago

Hi @mbloch,

I have tested the build and it fixes both this issue and the one reported in #519.

Thanks a lot for making mapshaper and also for being so helpful in fixing these issues!