search

aiddata / gcdf-geospatial-data

Repository for AidData's Geospatial Global Chinese Development Finance Dataset (GeoGCDF)
https://aiddata.org/china
Other
32 stars 8 forks source link

Can we simplify large GeoJSONs? #12

Open sgoodm opened 3 years ago

sgoodm commented 3 years ago

Currently there are 3 GeoJSONs that are about 7x larger than the next largest GeoJSONs.

48M latest/geojsons/178.geojson
48M latest/geojsons/56959.geojson
48M latest/geojsons/695.geojson

These 3 make up about 30% of the combined GeoJSON's total size.

Possible solution is to simplify these features, but when do we decide to simplify and how much do we simplify?

sgoodm commented 3 years ago

Worth noting that all 3 of those are the same feature (a lake with very precise mapping of edges, and tributaries, creeks, etc around it)

sgoodm commented 3 years ago

Also realized that the JSON write includes indents which we can drop to reduce file size

sgoodm commented 3 years ago

JSON write updated in 9dc4f1a

sgoodm commented 3 years ago

Confirming this is still an issue and these are pushing us close to GitHub max file size for combined geojson zip

sgoodm commented 3 years ago

Moving this off of launch milestone

sgoodm commented 3 years ago

Note that these trigger errors based on GDAL's OGR_GEOJSON_MAX_OBJ_SIZE env var, which defaults to 200MB

https://gdal.org/drivers/vector/geojson.html https://gis.stackexchange.com/questions/300419/ogr2ogr-import-fails-with-error-1-geojson-object-too-complex

sgoodm commented 3 years ago

One more note: considering that this project is for a hydro power plant, we can likely get a better feature than the entire lake.