azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
80 stars 26 forks source link

Read compressed augmented diffs #122

Closed mojodna closed 5 years ago

mojodna commented 5 years ago

Augmented diffs are currently loaded as uncompressed JSON from s3://<bucket>/<path>/<sequence>.json.

Gzipped JSON is now available, e.g. s3://osm-detroit/augmented-diffs/003/288/525.json.gz. This reduces the amount of storage these take on S3 and makes them more manageable to list.

AugmentedDiffSource should read these compressed versions instead.