Daily(?) updates for the vector tiles data (Berlin)

osmberlin / osm-parking-processing

Processing pipeline to generate data on public parking from OpenStreetMap-Data.

https://parkraum.osm-verkehrswende.org/project-vector-tiles/

GNU Affero General Public License v3.0

17 stars 2 forks source link

Daily(?) updates for the vector tiles data (Berlin) #12

Closed tordans closed 2 years ago

tordans commented 2 years ago

We have a map 🥳 . https://parkraum.osm-verkehrswende.org/project-vector-tiles/#16.5/52.513904/13.427681

One next step is, to get recurring updates going. Ideally hourly, maybe daily, maybe weekly…

Follow up to https://github.com/gislars/strassenraum-berlin/issues/10

gislars commented 2 years ago

just some thoughts:

for daily updates The current scripts can be run as cron job. The berlin-extract from download.geofabrik.de will be downloaded and processed. Since the extracts are generated once a day, we would need an other solution to update the data.
for updates less than a day For now, the scripts are run for a whole, fresh import and recreating all the tables. A continuously updated database can be setup with osm2pgsql. But for our post processing the way we update the data needs to be changed. For the future we might want to seed a cache with all the vector tiles. Currently we just use varnish and with every update we drop the varnish cache. So with an continuously updated database we would also like to update just a subset of vector tiles. It all depends on how fast the vector tiles can be (re)created. For this task we should have a look at https://github.com/onthegomap/planetiler

tordans commented 2 years ago

for updates less than a day

The PlanetTiler approach would be to not bother with partial updates and just re-run everything (but fast). I like the simplicity of that.

I understand now, we have two variables here

the max time it takes for the data to process the given region
the source of the data that we process, which also needs to be fresh (or running the script is pointless ;-))

My take: Let's look at the simplest (daily?) solution for now as a next step…

gislars commented 2 years ago

Jede Nacht wird nun die DB automatisch aktualisiert, d.h. neues PBF von Geofabrik heruntergeladen und mit unseren Scripten dann verarbeitet. Den Cache-Control-Header habe ich auf 12 Stunden gesetzt, d.h. nach dieser Zeit sollte der serverseitige Cache (Varnish) neue Daten von pg_tileserve holen.

joshinils commented 2 years ago

Wie sieht es mit #23 aus?

tordans commented 2 years ago

Ich denke, wir können dieses Ticket schließen. Vielen Dank für die (fast) täglichen Updates, @gislars.

Für mögliche Folge-Themen wie Error-Handling wenn es keine neuen Daten bei der Geofabrik gibt (etc.) sollten wir dann bei Zeiten ein Folge-Ticket erstellen.