search

owid / owid-grapher

A platform for creating interactive data visualizations
https://ourworldindata.org
MIT License
1.39k stars 229 forks source link

Replace `make sync-images` by fallback in Cloudflare worker #3328

Closed Marigold closed 7 months ago

Marigold commented 8 months ago

Currently, when creating a staging server or setting up dev environment, we have to sync all GDoc images from r2:owid-image-upload/production to r2:owid-image-upload-staging/my-env by calling make sync-images in order to make it work.

We could simplify this by having a Cloudflare worker handle requests to https://images-staging.owid.io/my-env/xxx.png and redirecting them to https://images.owid.io/production/xxx.png if the image doesn't exist. If the dev uploads an image to R2, the Cloudflare worker will use it instead of falling back to the production bucket. This would make the sync-images command and the need for rclone obsolete.

(We do it like this for Data API, and it works well. We could just copy the worker and change its config.)

@ikesau just to let you know that there's a possibility like that. Syncing images works well right now, so there's no need to change it. (Syncing images takes only ~30s so there's not much to gain either)

danyx23 commented 7 months ago

We'll probably want to switch to CF images in the not too distant future, so probably not worth spending time on this.