search

dart-lang / dart_ci

Tools used by Dart's continuous integration (CI) testing that aren't needed by Dart SDK contributors. Mirrored from dart.googlesource.com/dart_ci. Do not land pull requests on Github.
BSD 3-Clause "New" or "Revised" License
18 stars 5 forks source link

Create Firestore database backup and restore policies #75

Open whesse opened 4 years ago

whesse commented 4 years ago

Until Feb 2020, the Firestore database at dart-ci was not backed up externally, so database corruption could have resulted in permanent loss of all data.

We need a system for regularly backing up the database to cloud storage, and a verified procedure for restoring the database from the backup.

Additional systems are needed to restore results produced since the last backup, to bring a restored database up to date with current results.

whesse commented 4 years ago

Database has been backed up to cloud storage using the command

    gcloud firestore export gs://dart-test-results/backup/firestore-production/2020-02-11  --async --project dart-ci

This backup is done manually, at least weekly. No other work has been completed on this item, still urgent to do remaining work.

whesse commented 4 years ago

Restoring the database from backup, and replaying missed results to the database, can be done by restoring to the dart-ci-staging database. If we don't want to keep an entire second copy of the database, we could delete earlier data from dart-ci-staging, but the delete operations probably cost more than the storage cost of leaving that data in the copy. We should remove all data from the staging database, and ensure its indexes are identical to the indexes on dart-ci, before restoring to it.

whesse commented 4 years ago

Because the dart-ci-staging project is located in region europe-west3 and the dart-test-results bucket in dart-ci is located in region us-central1, we cannot restore backups from dart-ci into that bucket directly to dart-ci-staging. I have to use the Google Cloud Storage Transfer solution to copy the data into the default dart-ci-staging.appspot.com.

I have deleted the tables from staging, except the special collection called "staging" and the github-issue-tracker table, ensured the indices are the same as on the production firestore database, and run the import from the copied backup. It took about 30 minutes to restore from the 500 MB of backup files. I also had to recreate the collections (the records and indices were already there, the collection names just didn't show up), and make sure when creating them to create a first document with id like '00000000', so it would be first alphabetically and I could find it to delete it (It makes you create a first document, even though there are already documents there).

The commits_cache test worked successfully on production, and on staging (which was missing the commits it checked) the fetching and storing of commits from gititles worked correctly as well.

whesse commented 1 year ago

The results database in Firestore on dart-ci is backed up weekly to cloud storage by manually running commands like

gcloud firestore export --project dart-ci gs://dart-ci-backup/firestore-production/2023-09-04

Automating this backup would be good. We have never had to use this backup yet. Reducing the priority of this issue to P3.