search

owid / owid-grapher

A platform for creating interactive data visualizations
https://ourworldindata.org
MIT License
1.37k stars 229 forks source link

Migrations that remove columns break ETL #3535

Open Marigold opened 5 months ago

Marigold commented 5 months ago

ETL uses its own ORM. The problem is that if someone removes a column with a migration in grapher, ETL breaks. This happened recently with isBulkImport removal that broke fasttrack (cc @marcelgerber).

Any ideas on how to make it more resilient? (I don't have any yet)

danyx23 commented 5 months ago

Hey Mojmir! I was wondering if we could have a bot without too much trouble that checks prs in the owid-grapher repo and whenever there is a push checks if there were changes in ./db/migrations/. If so, then it adds this checklist as markdown to the PR description so that you are reminded to check all points and see if they apply.

What do you think?

Marigold commented 5 months ago

Posting a checklist to PR would be super neat! I plan to work on this soon, I'll make sure to add it.

marcelgerber commented 5 months ago

Hey Mojmir! I was wondering if we could have a bot without too much trouble that checks prs in the owid-grapher repo and whenever there is a push checks if there were changes in ./db/migrations/.

Just noting that this part of the proposal is super easy; see https://github.com/owid/owid-grapher/blob/dff12bf7e17aaf441e8c2a00a089df7c31c2c7dc/.github/workflows/sync-grapher-schema-to-digital-ocean.yml#L6-L7 for example.