owid / owid-grapher

A platform for creating interactive data visualizations
https://ourworldindata.org
MIT License
1.35k stars 227 forks source link

:sparkles: Reduce downtime of make refresh by swapping the database #3747

Open Marigold opened 3 days ago

Marigold commented 3 days ago

Running make refresh induces downtime of about ~2 minutes. This is annoying when refreshing staging servers. The idea of this PR is to create a _temp database, load it and then swap with the original. Unfortunately, there's no RENAME DATABASE command in MySQL and we have to rename it table by table (and recreate views).

See slack

Other potential solutions:

Note that owid user on staging cannot create a database. We have to change its privileges in ops to

mysql> GRANT ALL PRIVILEGES ON *.* TO 'owid'@'%';
mysql> FLUSH PRIVILEGES;
owidbot commented 3 days ago
Quick links (staging server): Site Admin Wizard

Login: ssh owid@staging-site-refresh-swap

SVG tester: Number of differences (default views): 1264 ([f285de](https://github.com/owid/owid-grapher-svgs/commit/f285deb54690ca80356ed7438b75a9363ca72ac9)) ❌ Number of differences (all views): 531 ([b8e7fe](https://github.com/owid/owid-grapher-svgs/commit/b8e7fe2b0cb1bd6ba9e5a19533e1dd65a1024bdc)) ❌

Edited: 2024-06-26 09:51:44 UTC Execution time: 1.12 seconds