Closed Marigold closed 1 week ago
I enabled general query log for all new staging servers. If something goes wrong, we should be able to check /var/log/mysql/query.log
for what happened and rollback in the worst case.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
It looks like something might be resetting MySQL database on staging servers. This happened two times already
By @pabloarosado
by @lucasrodes
This looks like
make refresh
has been run on those servers. There are only a couple of places we run it in Buildkitemaster
andmaster-1
branches on new commits to grapher repomaster-1
master
at midnightHypotheses:
To mitigate this, we could enable general log in MySQL on staging servers and start backing it up to disk.