backup/restore: Explain scheduled backup directory structure

[sheaffej]

John Sheaffer (sheaffej) commented:

I am frequently being asked by prospects and customers how they should interpret the directory structure created by scheduled backups. We should explain this directory structure in the docs.

Why do people care about the directory structure? Backups are critical to the production readiness of an application. Admins want to have complete confidence that the backups are correct, accessible, and that they can identify which backup is needed while under high pressure to restore a database that needs a previous backup. So they look at the folder where backups are written, and then they don't understand what they are seeing.

Also to restore a scheduled backup, the top-level URL used to create the schedule is not the same lower-level URL to perform a restore. This requires an admin who is performing a restore to understand the directory structure of a scheduled backup: See: https://github.com/cockroachdb/docs/issues/9999

Additionally, backup retention in CockroachDB is delegated to the administrator of the destination storage. This means in order to properly purge the right files for a set of backups, the admin needs to understand which folders can be purged (i.e. the day-level folders) and which folders should not (i.e. the top-level folder, year folder, month folder, etc)

Points to cover There are some key points about the directory structure we should cover.

• A "backup folder"(i.e. a folder containing a backup) is a folder with the BACKUP_MANIFEST file. If you try to restore by specifying a folder that does not have a BACKUP_MANIFEST file, the restore will not be able to find the backup.
• With scheduled backups, the folders are organized under <top-level-folder>/Year/Month/Day folders where only the top-level folder is specified when creating the backup schedule
• Incremental backups are placed as subfolders under the folder containing the full backup. In the case of daily fulls w/ hourly incrementals, the incremental backups are 3 folders below the full backup: <full-backup>/day/hour/<incremental-backup>
• Why does the day X full backup folder have incremental backup subfolders for day X and day X+1?
  • Per @mwang1026 : It puts the incremental under whatever the latest already completed backup is when the incremental kicks off, So if you are "next day" but that day's "full" hasn't finished, we kick off the incremental anyway to ensure coverage of backup time frame
  • Ref internal thread: https://cockroachlabs.slack.com/archives/C2C5FKPPB/p1612566374297100

Where should this go? I believe this explanation applies to all of the pages listed below. So perhaps we should have a separate page, and link to this new page in all of these related pages:

• https://www.cockroachlabs.com/docs/v21.1/create-schedule-for-backup
• https://www.cockroachlabs.com/docs/v21.1/manage-a-backup-schedule
• https://www.cockroachlabs.com/docs/v21.1/take-full-and-incremental-backups
• https://www.cockroachlabs.com/docs/v21.1/backup
• https://www.cockroachlabs.com/docs/v21.1/restore

It would be most useful if the page had an example of a backup's directory structure, and explained what the files and folders contain.

Jira Issue: DOC-1576

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB docs!