:sparkles: create temp database in make refresh and swap it at the end

owid / owid-grapher

A platform for creating interactive data visualizations

https://ourworldindata.org

MIT License

1.35k stars 227 forks source link

:sparkles: create temp database in make refresh and swap it at the end #3684

Closed Marigold closed 3 weeks ago

Marigold commented 3 weeks ago

I'm having issues with refreshing MySQL on our staging servers. To keep the data up to date, we have to refresh MySQL after every commit to the ETL. The problem is that there might be a grapher baking process going on at the same time, and it will fail because MySQL is refreshing itself.

This PR makes the refresh process more "atomic" by creating a temporary database first and swapping it at the end. If this doesn't work, we'll have to start locking the database everywhere, which would be annoying.

owidbot commented 3 weeks ago

Quick links (staging server): Site	Admin	Wizard

Login: ssh owid@staging-site-mysql-refresh-swap

Site-screenshots: https://github.com/owid/site-screenshots/compare/mysql-refresh-swap
SVG tester: https://github.com/owid/owid-grapher-svgs/compare/mysql-refresh-swap

SVG tester:

Number of differences (default views): 0 ✅ Number of differences (all views): 0 ✅

Edited: 2024-06-05 08:28:53 UTC Execution time: 1.15 seconds

Marigold commented 3 weeks ago

Not possible because owid user doesn't have privileges to create a new database.