search

hotosm / osm-analytics-cruncher

Backend code for osm-analytics
https://github.com/hotosm/osm-analytics
15 stars 10 forks source link

Add historic data for 2017, 2018 #14

Closed cgiovando closed 6 years ago

cgiovando commented 7 years ago

The historic snapshot for 1/1/2017 is missing from the timeline in the before/after tool.

Let's also document the process of creating these snapshots to make it easier to add them for other map features.

jenningsanderson commented 6 years ago

:wave: Hey @cgiovando I've updated the historic snapshots, the 2017-01-01 snapshot is available here: https://s3.amazonaws.com/mapbox/osm-qa-tiles/historic/2016-Q4-qa-tiles.mbtiles

See: http://osmlab.github.io/osm-qa-tiles/historic.html# for more information on these files / how they were generated. Note the move to quarterly snapshots instead of annual snapshots; for better historical resolution, these historical snapshots should be able to plug directly into the current workflow.

tyrasd commented 6 years ago

We also had incomplete (and in a few cases wrong) historic osm-analytics snapshot data (e.g. for waterways layers only two snapshots had been calculated that contained odd results, and at least one of the highways snapshots had been produced for the correct timestamp).

I'm now re-creating all the historic osma-snapshots using the osm-qa-tiles quarterly historic snapshots (using the Q4 snapshot of each year) in order to fix the bad data in the current historic snapshots and to provide the missing more recent yearly snapshots for 2017 and 2018. The resulting snapshots until 2017 (inclusive) are already online; 2018 is still processing but should be done in a few hours //edit: as are the 2018 snapshots

document the process of creating these snapshots

In principle, it's just the same as invoking the normal cruncher, but using a different osm-qa-tiles input file (i.e. one of the quarterly snapshots instead of the most recent planet osm-qa-tiles file) specified by the appropriate URL as OSMQATILES_SOURCE environment variable. ( One minor adaption is needed because the historic osm-qa-tiles snapshots are not gzip-compressed: just remove the | gzip -d command in run.sh).