search

ASFHyP3 / hyp3-gamma

HyP3 plugin for generating SAR products with GAMMA
BSD 3-Clause "New" or "Revised" License
30 stars 7 forks source link

resolve "GeoJSON object too complex" error in create_water_mask #401

Closed asjohnston-asf closed 1 year ago

asjohnston-asf commented 1 year ago

Here's a crude and quick fix to the ERROR 1: GeoJSON object too complex, please see the OGR_GEOJSON_MAX_OBJ_SIZE environment option errors we see when processing these scenes over the Aegean Sea:

The OGR_GEOJSON_MAX_OBJ_SIZE setting is specific to the GDAL GeoJSON driver, documentation is at https://gdal.org/drivers/vector/geojson.html#configuration-options

We're using GeoJSON for the temp file out of convenience, we might be able to avoid this issue by using any other GDAL-supported vector format.

Alternatively, if we stick with GeoJSON, it would be nice to set OGR_GEOJSON_MAX_OBJ_SIZE based on the size of the temporary geojson file instead of to an arbitrary big value.

@jacquelynsmale any thoughts on those options? Would you be interested in taking a stab at implementing either? I've only checked that this code still passes the unit tests, I haven't yet verified it fixes the issue for the specific pair in question.

hjkristenson commented 1 year ago

@asjohnston-asf Is there any reason why we don't just use the ESRI-shapefile format? That's what the shorelines file is, and I see that it's supported in the list that you linked. Is it just that the extracted portion of the shapefile is rendered in GeoJSON right now? Is it much harder to just keep it in the shapefile format?

asjohnston-asf commented 1 year ago

That was my first thought, my only (very minor) hesitation is that format is multiple files, so we might have to tweak the temp file details to clean up the temporary sidecar files.

hjkristenson commented 1 year ago

That makes sense. I guess we could test the bumped-up GeoJSON approach to see if it does the trick, and try out the shapefile approach if there are issues (or concerns about what it would take to find an appropriate object size limit).

asjohnston-asf commented 1 year ago

I've confirmed the trufflehog failure is a false-positive related to story map URLs already merged to develop.