Open cweagans opened 2 years ago
We had this feature a very long time ago, but it was quite broken - it expected every plugin to write it's backup contents to stdout and then tar'd it all up, which ended up causing broken backups in some cases.
This is actually a bit more complex than you'd expect.
I'd previously tried to raise money via Github Sponsors for backup/restore functionality across the project, but didn't get to my goal of $500. That said, if you'd like to work on this functionality, by all means go ahead. I'd be happy to provide feedback once you have a skeleton working to get it to a place that is usable by core, datastore, and community plugins.
Dokku is more or less at the $500 a month goal, so I'll start working on this in the next few releases. I think there is still some work to move a few files out of the git repositories (docker options, env vars, nginx configs) but those should be relatively straightforward to do.
Let me think about a good backup interface and then post my comments here. I'm guessing a few things will change about how datastores do backups, but I believe that will be a Good Thing™.
Okay I think what I'll do is something like this:
backup:export [--app $APP] [--service $SERVICE] [--backup-dir TMP]
This new command will create a tar.gz
file in the /tmp
directory with the desired data. The /tmp
dir may be overriden via the --backup-dir
flag, and the file will be output there if so. Dokku will attempt to write a temporary file to the --backup-dir
, and if it fails, the backup will exit immediately.
The backup will be buffered to a directory in /tmp
while being generated. Users are expected to have enough space in /tmp
to hold backup data.
If no app/service is specified, the backup will contain all apps and all services.
The full path to the tarball will be written to stdout.
Log messages for backup:export
should always appear on stderr
.
backup-global-export $PATH
: New trigger for backing up global settings for a plugin. The plugin can create a directory in that path and export whatever it wants to that directory. I would suggest plugins create a data
and a config
dir, and place their files appropriately.backup-app-export $APP $PATH
: New trigger that takes an app name and a path. The plugin can create a directory in that path and export whatever it wants to that directory. I would suggest plugins create a data
and a config
dir, and place their files appropriately.backup-service-export $SERVICE_TYPE $SERVICE_NAME $PATH
: New trigger that takes a service type (eg. mysql
), name and a path. The plugin can create a directory in that path and export whatever it wants to that directory. I would suggest plugins create a data
and a config
dir, and place their files appropriately.pre-backup-export $PATH $BACKUP_FILE
: Will allow plugins to do things prior to the export (though likely unnecessary).pre-backup-app-export $APP $PATH
: Will allow plugins to do things prior to the export (though likely unnecessary).post-backup-app-export $APP $PATH
: Will allow plugins to do things after to the export (though likely unnecessary).pre-backup-service-export $APP $PATH
: Will allow plugins to do things prior to the export (though likely unnecessary).post-backup-service-export $APP $PATH
: Will allow plugins to do things after to the export (though likely unnecessary).post-backup-export $PATH $BACKUP_FILE
: Will allow plugins to do things after to the export (though likely unnecessary).backup:import <backup-file-path> [--app $APP] [--service $SERVICE]
This new command will take a backup file path and an optional app/service name. If no app/service is specified, it will attempt to import everything.
This command may be destructive, and users are expected to take caution when running it. Confirmation of restoring on top of existing data will occur.
When restoring, if an app or service is not specified, the entire backup is first extracted before being restored. Thus, users should have enough space for 2x the size of the extracted backup.
Restores aren't guaranteed to work in all cases - particularly if dependencies are missing or there are external issues like a docker bug. Users will be encouraged to regularly test their backups on new servers.
backup-global-import $PATH
: New trigger, but for imports. Plugins should not do anything here other than import files. No restarts or anything should be triggered as the import may not be complete.backup-app-import $APP $PATH
: New trigger, but for imports. Plugins should not do anything here other than import files. No restarts or anything should be triggered as the import may not be complete.backup-service-import $SERVICE_TYPE $SERVICE_NAME $PATH
: New trigger, but for imports. Plugins should not do anything here other than import files. No restarts or anything should be triggered as the import may not be complete.pre-backup-import $PATH $BACKUP_FILE
: Will allow plugins to do things prior to the importpre-backup-app-import $APP $PATH
: Will allow plugins to do things prior to the importpost-backup-app-import $APP $PATH
: Will allow plugins to do things after to the importpre-backup-service-import $APP $PATH
: Will allow plugins to do things prior to the export, like ensure the service doesn't already existpost-backup-service-import $APP $PATH
: Will allow plugins to do things after to the export, like start the servicepost-backup-import $PATH $BACKUP_FILE
: Will allow plugins to do things after to the import, like rebuild all appsThat sounds great! If I was going to extend this functionality to e.g. store backups on S3, would I use the post-backup and pre-import hooks? How would that plugin prune old backups? (Totally fine if the last bit is just a custom command or something)
I think I would like to concentrate on this creating a tarball on disk. If someone wants to upload that somewhere, they can do so - similarly, if their backup process should prune old backups, they can handle that themselves.
To extend and do something custom with the file, you'd use a hook of some sort.
Absolutely -- not suggesting that s3 storage be implemented in Dokku directly. Just thinking through the ways that this could be extended by other plugins -- if the answer is that s3 storage stuff should be done without interacting with the backup plugin directly, that's totally workable.
I just want to avoid the issue we have with the datastore plugins where each one implements S3. Folks use things other than S3, so it makes us seem like users cant do something else.
I'd rather we document tools in the ecosystem to make those things possible, maybe even writing a tutorial for them.
Totally understood! Appreciate the time and thought :)
Description of problem
As a dokku admin, if I want to export a backup of an entire application and all of the dependencies, I need to know how that application works, whether or not the plugins that provide the backing services offer backup commands, etc. It would be much better if there were a mechanism by which an application and all of the attached services can be told to export whatever they need to export.
Possible Solution
I think a very lightweight solution could be put in place here: we just need six new plugn triggers:
The only thing Dokku needs to handle is dispatching those triggers + providing a directory that each plugin should write stuff to or read stuff from (as well as backing up/restoring core data, of course -- from what I can see, this is already handled in other commands so it would just need wired up).
Example:
If I'm running an application that was pushed to Git with a MySQL backing service and a persistent storage directory, here's what I'm imagining would happen when I trigger a backup:
/tmp/backup/app-name-[timestamp]/
/tmp/backup/app-name-[timestamp]/code.tar.gz
/tmp/backup/app-name-[timestamp]/domains.txt
/tmp/backup/app-name-[timestamp]/some-meaningful-name.tar.gz
[app-name]
into dir/tmp/backup/app-name-[timestamp]
mysqldump
ormysqlhotcopy
or whatever. The implementation details don't matter and will vary by plugin.When I ask to restore something, the reverse should happen. If I have an S3 backups plugin or w/e, it should be responsible for downloading the backup that I've specified and extracting it into the location where Dokku expects the backup data to be available. Each plugin is responsible for reading its own data from the backup dir.
Aside
I feel like this existed at some point and that I'm reproducing it (or something like it) from memory, but maybe I'm remembering wrong? Is there some reason that the core product should not provide this kind of plumbing? Backup and restore seems really important for a PaaS.