Backup and restore

[danielbachhuber]

As a WordPress developer, it would helpful if I had an easy way to snapshot and restore a WordPress site. Specifically:

• A snapshot should contain all user data (database, media files, themes and plugins), and a reference to the core version.
• A snapshot should also contain a manifest file (e.g. https://github.com/runcommand/manifest) so that it's possible to inspect what's inside the snapshot without restoring it fully.
• I should be able to push the snapshot to an external destination (e.g. Dropbox, Google Drive, or S3), and then list snapshots available in that destination.

[danielbachhuber]

From Nick Cernis via email:

I see you have plans for a snapshot command, but would love to see that extended to files as well as the database as a way to backup and migrate sites:

wp snapshot – take a full snapshot of the wp-content folder and database, packaged as a single snapshot archive/zip.

wp snapshot list – show a list of any snapshots stored in the snapshot directory.

wp snapshot restore [snapshot-id or path to local/remote snapshot file] – restore the snapshot, overwriting the existing wp-content directory and database.

wp snapshot restore [snapshot-id or path] --search='example.com' --replace='example.net' – restore the snapshot and use wp search-replace simultaneously. (Handy for migrations or recreating a remote site locally; would be amazing to run wp snapshot on a remote site and then recreate it locally with one command, using the path to the remote snapshot file stored on the live server.)

wp snapshot schedule --period='5m' --max='10' --exclude='uploads' – take automatic snapshots every 5 minutes, saving a maximum of 10 snapshots, excluding the uploads directory. (Useful for local dev when doing anything experimental/dangerous that involves the DB, but could potentially also be used as a backup option for hosts.)

[miya0001]

wp snapshot server to run php -s. 😎

[TheLastCicada]

What would be really slick is if it only backed up files in wp-content that were new or modified since the previous snapshot. The backup on the entire uploads directory could be enormous, but doing a master snapshot with differential changes could be an efficient way to do this. I like having a flag to enable or disable including files in the command - similar to the --exclude in the example.

[danielbachhuber]

What would be really slick is if it only backed up files in wp-content that were new or modified since the previous snapshot. The backup on the entire uploads directory could be enormous, but doing a master snapshot with differential changes could be an efficient way to do this.

Definitely!

Worth noting that it would introduce a degree of complexity, as you'd need to:

1. Perform a full backup every so often as a starting point for restoration.
2. Have the restore mechanism know how to restore from the full backup, then apply each of the incremental backups.

Out of curiosity, how are you doing backups now? In what ways could this WP-CLI command save you time and effort?

[TheLastCicada]

Combo of wp db export and rsync/s3cmd. We've started leaning heavily on the S3 versioning and lifecycle rules, and they essentially provide a way to do snapshots and retention rules without having to think much about the mechanics of it. Which makes me think.... if there was an easy way to integrate S3 with wp-cli where someone could provide their access key and secret key, then have wp-cli use S3 as an endpoint for backups, that'd be pretty sweet. Could even do the bucket provisioning and lifecycle/versioning rules definitions via the plugin..... like wp backup initiate --destination=s3 --bucketname=mybackuplocation --keep-for=365d --all-versions --glacier-archive-after=90d --backupname=mydailysitebackup. Then you could run in a system cron job wp backup run mydailysitebackup. I envision the daily run doing something similar to what s3cmd sync does. Databases you'll want to keep the full daily backup for, which, once compressed with gzip, are usually reasonably space efficient. Worth noting that we've started using https://github.com/mishudark/s3-parallel-put for these since s3cmd is single threaded and will take days and days on sites with a large number of files. The AWS CLI tool may offer some gems for this as well. As you mentioned, you can have different backends for this like Google Drive, Azure Blob Storage, FTP, etc. They'd all need pretty custom rules as they all behave differently.

My overall strategy with backups is to rely on other tools as much as possible so the logic we have to write is minimal. When you get too clever with your backup strategies, it is easy to miss an edge case and not have a valid backup when you need it. I'd rather have my backups be more dumb and pay for more cheap storage than risk missing something.

Another nice tool we use is https://sourceforge.net/projects/autobackupmysql/, which handles the file management if you want to set rules like "keep daily backups for 30 days, monthly backups for 2 years, and yearly backups for 10 years". This is very helpful for database backups.

[tedmasterweb]

Ok, jumping in late here and probably should have read this before posting my feature wish #25 but you might find a bash script I wrote (lot's of room for improvement, please go easy on me…) helpful in reaching this goal

[niladam]

hey @tedmasterweb -- i'm working on my own version of a really, simple backup shell script, and i like some functions in yours. Can i borrow and adapt some of them ? -- This will mean i'll clone your current bitbucket repo to one on github -- mentioning you as the source.

[tedmasterweb]

Hi @niladam -- Please use whatever works for you and share it somewhere. That's fine. Please include a link back to our company (should be in the readme / license, I think) and let me know where I can see your work. Maybe I can lend a hand.

BTW, there are about 4 places where I should be using wp-cli in that script. If it were up to me and I had time, I would start by doing that as the GREPs I'm using in that script for search and replace are painful, to say the least. Good luck!

[niladam]

@tedmasterweb My script is already up and currently functioning. Still in development though, but it's a full working script. You can have a look at wpeb to see how it works and looks like.

Thank you for your permission, i'll mention you in the source code and in the readme too.

[niladam]

hey @tedmasterweb i've forked your repo into fork-of-tedmasterweb-wp-backup-and-restore; Thanks again for your permission.

[tedmasterweb]

Excellent and you're welcome! Let's see if we can fold some of that good work back into this project at some point too…

[aj-adl]

Just jumping in to say

1) this would be a fantastic edition to WP CLI. 2) Personally I'd get a lot of use out of a 'state only' option - ie backup uploads + db, as everything else is part of a git repo (theme) or handled via composer (plugins, libs etc).

At the moment our backups are usually just uploads, dependency config files (package.json, yarn.lock, composer.lock ) and the DB.

I can imagine there being a lot of different scenarios where people will want to exclude certain folders or include certain folders - allowing for some kind of yaml / json / config file would solve all those problems.

[danielbachhuber]

An idea to include from Marco Arment:

Automatically test your database backups

One of the issues of yesterday’s GitLab.com “database incident” is that most of their database backups weren’t being tested, and when they needed a restore, they discovered that most of the backup methods hadn’t been working.

Untested backup methods that turn out to be missing or broken are extremely common. I can’t fault them much because it’s a very easy mistake to make: most backups, by nature, never need to be restored from, so you never realize if something changes and they stop working… until it’s too late.

The solution is to frequently and automatically test backups

It would be neat if the command facilitated this.

[mbovel]

I've been thinking about a backup/restore strategy for some WordPress sites I host lately and couldn't find an elegant solution. I am happy to see that other people have the same need and I would love to see such a feature in WP-CLI! Here are some random thoughts I had.

1. The uploads folder can be backup as is.
2. We already have a variety of tools to export the database: wp db export, wp search-replace as explained in #26 or directly mysqldump for example.
3. The rest (WordPress core, themes, plugins, etc.) is really where we miss something.

For this last part, maybe stating the obvious, there are two approaches:

1. Backing up the whole WordPress folder.
2. Backing up configuration (list of plugins as opposed to the plugins files for example).

For some reasons, lot of existing backup solution for WordPress use the first approach. But given that WP-CLI already has all the tools to impose specific versions of WordPress core, plugins and themes, the second approach would seem way more elegant to me.

All we need would be a slightly modified version of wp plugin list (wp plugin backup for example) that would write the list of active plugins and their version to a file (plugins.yml for example), and a corresponding command that would be able to restore them (wp plugin restore for example). The same could apply to themes and core.

Personally, that's all I need. I could then backup or version it the way I want (using rsync, git, or whatever).

I would really like such a modular approach (I think this is the direction @danielbachhuber was going with Dictator) as it would fit easily more workflows. An additional shortcut command like wp snapshot that does all of this at once and maybe provides more features like zipping would of course be welcome too, but not as the single option.

As far as I know how WP-CLI works, implementing these features that way would be really simple.

About incremental backups, that seems like way more work to implement it in the right way and I'm not sure this is really WP-CLI job as we already have general tools for that and that a WordPress specific way would maybe be better implemented as a plugin.

[erikjoling]

I agree with @mbovel

For a snapshot the plugin, theme and core-files are redundant. Their information should be stored in configfiles. That way the size of the backup will be kept at minimum (thus better portable) while all data could be restored.

[viktorbijlenga]

I think it would be a good idea to have the theme included in the backup, since it can't be restored always from an other source (compared to core or a plugin).

[erikjoling]

Why would a theme be different than a plugin? As far as I'm concerned all WordPress source-code should be hosted somewhere, themes included.

[viktorbijlenga]

A more complete backup will always be a good idea. Plugins and core has better redundancy since they all are forced to use version control. Themes is a lot more messy, since the source is more or less using their own way of backup, sometimes there are no backup at all.

[mbovel]

I made a small prototype to illustrate some of the ideas I emitted in my previous message: wp-backup-restore-command. This is a WP-CLI command that enables backing up core files, database and plugins from a WordPress setup. Themes support could be implemented exactly in the same way as plugins.

This command does not copy any files when backing up, it only generates JSON configuration files and SQL dump. It is then up to the user to save these files and the uploads folder with their favorite versioning tool.

An example setup that uses this command in conjunction with git to enable easy rollback to a previous version can be found here: example-wp-setup.

I hope this work can be useful to move the discussion forward and I would love to hear your feedback.

[danielbachhuber]

I made a small prototype to illustrate some of the ideas I emitted in my previous message: wp-backup-restore-command.

Neat!

Another idea to throw in the mix: only store the original versions of media files (e.g. ignore thumbnails), and the defined media sizes. Then, upon restore, regenerate the thumbnails.

[abhijitrakas]

This command tries to accomplish some of the above features, might help someone.