geotrellis / vectorpipe

Convert Vector data to VectorTiles with GeoTrellis.
https://geotrellis.github.io/vectorpipe/
Other
74 stars 20 forks source link

Read partitioned, compressed augmented diffs #62

Closed mojodna closed 5 years ago

mojodna commented 5 years ago

AugmentedDiffSource currently assumes that augmented diffs (produced by https://github.com/mojodna/overpass-diff-publisher) are available by sequence number as uncompressed GeoJSON in the root of the target. This is currently implemented as https://github.com/azavea/osmesa/blob/4a1f3fd9c22cc02172805729b91036f35d66bfd8/src/common/src/main/scala/osmesa/common/sources/AugmentedDiffSource.scala#L30-L67

https://github.com/mojodna/overpass-diff-publisher/commit/309043fd9a86235c7ce745bcc60e003f35ffb762 and https://github.com/mojodna/overpass-diff-publisher/commit/911a3c7ef514f6fcd30d7545549afd4c85612776 changed overpass-diff-publisher's behavior so that it writes gzip-compressed data to partitioned paths (i.e. 000/000/000.json.gz instead of 0.json). (It currently writes to both keys for compatibility.)

To allow augmented diffs to be stored and listed more efficiently, AugmentedDiffSource should read from the partitioned, compressed versions of the sequences. Once this is done, the secondary write path in overpass-diff-publisher can be removed and existing, uncompressed diffs removed.

mojodna commented 5 years ago

This was done in #89.