OpenStreetMap data in Athena is not updated

srividyacb commented 2 years ago

The OpenStreetMap data hosted and used as external table in Athena is not updated since September 5th. The max timestamp of the planet_history, planet and changeset tables are of September. According to the registry, the data update is supposed to happen weekly.

Can someone help in taking look at this issue?

mojodna commented 2 years ago

There have been a few changes with planet.osm.org that need to be reflected in this pipeline:

[x] planet.osm.org uses a Let's Encrypt certificate, and the CA has expired in old Lambda runtimes, including the one used here. fixed by manually updating the runtime to Node 14.x.
[ ] planet.osm.org is no longer hosting some PBFs (including the Planet from 9/27); need to switch to a mirror
[x] torrent files aren't accounted for correctly, so even when content is mirrored correctly, Batch is attempting to transcode the wrong files

In addition, apex (the tool previously used to deploy Lambda functions) is no longer supported, so deployment should migrate to SAM.

mojodna commented 2 years ago

While the other issues should still be dealt with, updating the Lambda runtime unblocked the mirroring + transcoding jobs. Changesets are fully synced; the planet and history files are being transcoded now and should be fully updated in the next 12 hours.

mojodna commented 2 years ago

Everything is back in sync.

mojodna / osm-pds-pipelines

OpenStreetMap data in Athena is not updated #27