Something might be resetting MySQL database on staging servers

[Marigold]

It looks like something might be resetting MySQL database on staging servers. This happened two times already

By @pabloarosado

is it possible that this staging server has been reverted to master at some point in the last 6 days? I have the memory of having spent 1 or 2 hours replacing instances of "caloric intake" -> "calorie supply" in multiple charts, and now I don't see those changes. I did make some changes in metadata, but I think I also changed some FASTT. Could this have happened somehow? I'm puzzled

by @lucasrodes

mmmh ー I just experienced the same. I was working yesterday on drafting some charts, which I shared in this comment. I went back now to the stging server and the charts are gone. unsure if related, but today i re-ran a job in master to see if that solved Pablo’s issue. here the ci/cd job: https://buildkite.com/our-world-in-data/etl-automated-staging-environment/builds/5306 could it be that i triggered something that overwrote stuff on staging servers?

This looks like make refresh has been run on those servers. There are only a couple of places we run it in Buildkite

• We refresh MySQL for master and master-1 branches on new commits to grapher repo
• When syncing master-1
• In CRON on master at midnight

Hypotheses:

• It doesn't run on master only, but might be run on other branches too
• Manual triggers from Buildkite (restarting the job and selecting the wrong branch)

To mitigate this, we could enable general log in MySQL on staging servers and start backing it up to disk.

[Marigold]

I enabled general query log for all new staging servers. If something goes wrong, we should be able to check /var/log/mysql/query.log for what happened and rollback in the worst case.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.