Fixes #17036 - Adding katello-backup and restore for a proxy

[johnpmitsch]

See comment below for review instructions

[cfouant]

@johnpmitsch - aren't we calling capsule 'foreman-proxy-content' now? I definitely would not have a --capsule flag if that is the case

[johnpmitsch]

@cfouant yes, that is true, I can change it. I'll add it to the to-do

[cfouant]

@johnpmitsch - jumped the gun, just noticed the WIP tag, so thanks!

[johnpmitsch]

To Review

(fpc = foreman-proxy w/ content)

We are going to essentially "clone" a fpc using the hostname change and katello-restore

1. Spin up katello + external fpc (will call this fpc1)
2. Associate fpc to organization/lifecycle env and sync some repositories to it
3. Run katello-backup on fpc1
4. Spin up a new external fpc, don't sync (will call this new machine fpc2)
5. Copy over certs tar that was generated for fpc1 (should still be on katello server) to fpc2
6. Run katello-change-hostname to change fpc2 to the hostname of fpc1, you will have to use --certs-tar option and specify fpc1 certs tar
7. Copy over fpc1 backup to fpc2 and run katello-restore
8. Make sure content is actually restored on fpc2 using pulp-admin
9. Delete entry for fpc2 in katello and change /etc/hosts to point fpc1 hostname to fpc2 ip address
10. Sync fpc2 to make sure all content and functionality is restored
11. Sync new repos to fpc2, delete repos and re-sync, check content is there with pulp-admin
12. Register host to fpc2 and make sure content can be delivered to host

Testing Scenario Checklist

Since quite a bit is refactored in this PR, we have to test the old functionality is still there, here is a list of new and old scenarios to test, please check them off as they are completed and feel free to add more:

• [x] fpc clone using standard backup (using above process) (checked by john - worked fine)
• [x] fpc clone using online backup (using above process) (checked by andrew -worked fine)
• [x] katello-change-hostname works as standalone operation (can change hostname of single fpc with new certs tar) (checked by john)
• [x] katello-restore works as a standalone operation (can restore content on same fpc as backup) (checked by john)
• [ ] standard katello backup and restore still works (checked by john, cfoo is double-checking)
• [ ] online katello backup and restore still works (checked by john, cfoo is double-checking)
• [x] provisioning features work fine (checked by john, can confirm that the correct files are restored based on feature specified)

reminder: snapshots are your friend! :smile:

[akofink]

Online backup/restore using the steps provided works great. Confirmed on Centos 7 running Katello 3.4 with a RHEL7 client. APJ from me!

[johnpmitsch]

@ehelms up for re-review, I tried to keep the newer changes in a separate commit (will squash at the end)

[johnpmitsch]

[test]

[johnpmitsch]

In the current state of the PR, the user can't restore a logical backup without removing files from it (a logical backup performs both a standard and online backup, so both sets of files are present). The user is presented with this message when trying to restore a logical or incorrect backup

---- The given directory does not contain the required files or has too many files ----
---- All backup directories contain: config_files.tar.gz, pulp_data.tar* ----
---- An online backup directory contains: mongo_dump, candlepin.dump, foreman.dump, pg_globals.dump ----
---- An offline backup directory contains: mongo_data.tar.gz, pgsql_data.tar.gz ----
---- *pulp_data.tar is optional ----
---- Please choose a valid backup directory ----

They can move files depending on the type of backup they want to restore. Alternatively, we select one way of restoring a logical backup, or we add flags in to give the user a preference of which backup they want to restore when a logical backup is present. Are there any preferences/opinions on which way is best?

[cfouant]

I would prefer if we allow a user to keep their backup files in one place. Since doing a logical-db-backup will create both an offline and online backup, I think we should by default use the offline backup files, and if user wants to override and prefer the online files, we could create a flag in katello-restore

[jlsherrill]

Could someone define 'logical backup' ?

[johnpmitsch]

@jlsherrill it basically dumps all the things

--logical-db-backup          Also dump full database schema during offline backup

You will have files from a standard and online backup present when using the flag --logical-db-backup

[johnpmitsch]

@jlsherrill here are the files in a logical backup:

[root@centos7-katello-3-4 ~]# ls /backup/katello-backup-logical/
candlepin.dump  config_files.tar.gz  foreman.dump  metadata.yml  mongo_data.tar.gz  mongo_dump  pg_globals.dump  pgsql_data.tar.gz  pulp_data.tar

[cfouant]

@jlsherrill - Customer requested to add online schema in case the offline db files are corrupted. http://projects.theforeman.org/issues/18925

[jlsherrill]

How about we default to the offline (telling the user we are using the offline version, and how to remove/move files to use the online), and then if there is some corruption they can switch to the online by deleting (or moving the offline files).

This is both simple from a logic perspective, simple from a user perspective, but still lets the user 'choose' if they really have a need to.

[johnpmitsch]

@jlsherrill thanks, we'll go with that unless @cfouant has any objections

[cfouant]

@jlsherrill @johnpmitsch I'd prefer a --use-online-files or something along that line instead. This way the user can 1) keep all files together 2) if there was a corruption instance, it would be easier to support and diagnose with both file types available.

[johnpmitsch]

@cfouant That is a good idea, but I would argue that adding that functionality is a bit outside this PR. Since the current functionality when restoring a logical backup is that it will use the standard backup (correct that if I am wrong), @jlsherrill's method will keep things the way they are, but just add output to clarify that a standard backup is being used.

[cfouant]

@johnpmitsch I'm fine to keep it the way it is already with just a note stating the offline backups are being used for restoration; I would not make any notes about moving files around to perform an online backup.

[johnpmitsch]

@cfouant +1

[johnpmitsch]

@ehelms updated according to your comments

[cfouant]

@johnpmitsch - initial tests including snapshots are working! I don't think you'll need to hide the snapshot feature!

[johnpmitsch]

@cfouant Snapshots on a foreman-proxy are working on my end too :+1: I'm running tests again to make sure nothing was broken in the small updates I have made, other than that, waiting on you and @ehelms to approve!

[cfouant]

I'm getting the error of 'too many files' when trying to restore a backup created with --logical-db-backup:

There are too many files in the backup directory, please check only the required files are present.
---- The given directory does not contain the required files or has too many files
---- All backup directories contain: config_files.tar.gz, pulp_data.tar*
---- A foreman proxy content backup directory contains: mongo_data.tar.gz
---- A foreman proxy content backup directory contains: mongo_dump
---- A logical backup directory contains: mongo_dump, candlepin.dump, foreman.dump, pg_globals.dump, mongo_data.tar.gz, pgsql_data.tar.gz
---- *pulp_data.tar is optional
---- Please choose a valid backup directory
/usr/bin/restoretest:179:in `display_backup_options': undefined local variable or method `optparse' for main:Object (NameError)
    from /usr/bin/restoretest:119:in `backup_invalid_message'
    from /usr/bin/restoretest:213:in `backup_valid?'
    from /usr/bin/restoretest:239:in `<main>'

[cfouant]

Also, I see what you were saying about needing to instantiate optparse in katello-restore. I will look into that, because it should be available to that method.

[cfouant]

ACK from me, pending the update to the restore method (we agreed above to instead of reject a backup directory with 'too many files' to use the offline files as preference with a note to the user), and making optparse an instance variable again (my bad). I've tested all current methods on a capsule and on a main katello server. Works great! Thanks so much for your work on this!

[johnpmitsch]

@cfouant I'm able to run a restore of a logical backup, can you make sure the branch is up to date and if so, do an ls of the directory to see if there is any difference?

[root@centos7-katello-3-4 katello-packaging]# ls /backup/katello-backup-20170621134718 
candlepin.dump  config_files.tar.gz  foreman.dump  metadata.yml  mongo_data.tar.gz  mongo_dump  pg_globals.dump  pgsql_data.tar.gz  pulp_data.tar
[root@centos7-katello-3-4 katello-packaging]# ./katello/katello-restore /backup/katello-backup-20170621134718/
WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Are you sure(Y/N)? y
Starting restore from /backup/katello-backup-20170621134718: 2017-06-21 13:52:59 +0000
<snip>
Logical backup detected, using the standard backup files to restore
<snip>
Done with restore: 2017-06-21 13:58:10 +0000

[johnpmitsch]

@cfouant updated optparse

[johnpmitsch]

@cfouant updated to remove the unnecessary katello-service start, all retesting is passing on my end

[cfouant]

ACK @johnpmitsch great job, wonderful work!