Implement an EBS snapshot backup functionality

[GoogleCodeExporter]

1. To which tool/application/daemon will this feature apply?

Tungsten Replicator

2. Describe the feature in general

When enabled, a backup will be taken using EBS snapshots. There will be several 
requirements in order to use this:

* The MySQL data directory must be part of a different EBS mount than root or 
Tungsten
* API credentials must be declared in a configuration file on each host
* sudo access is required to unmount/mount devices

3. Describe the feature interface

When enabled, tpm will validate the requirements are met. If there are no 
issues, the interface will be the same as other backup methods.

We may need to support an option that will turn off replication and stop mysql 
prior to taking the snapshot. If the dataset is fully InnoDB then this step is 
not required.

4. Give an idea (if applicable) of a possible implementation

The EBS snapshot support should be developed as a child of a parent snapshot 
backup method. There are some standard concepts that will be useful with other 
storage providers.

On backup
* Start the snapshot
* Watch for completion of the snapshot sleeping for 5 minutes between checks
* Tag the snapshot with a cluster name, machine ID and tungsten tag so it can 
be discovered later
* Store the snapshot ID and region as part of the on-disk backup file

On restore
* Check that the snapshot exists and is in the same region
* Copy the snapshot to the local region if needed
* Create a new EBS disk from the snapshot in the appropriate AZ
* Put the replicator offline
* Stop MySQL
* Unmount the old disk
* Deassociate the old EBS image
* Associate the new EBS image
* Mount the new disk
* Start MySQL
* Put the replicator online
* Delete the old EBS image

On tungsten_provision_thl

This needs discussion. It will be very simple to just use the `trepctl backup` 
and `trepctl restore` commands since the created backup file is small. There is 
little benefit to the advanced work done with mysqldump/xtrabackup to create 
the backup file directly on the target host.

5. Describe pros and cons of this feature.

5a. Why the world will be a better place with this feature.

5b. What hardship will the human race have to endure if this feature is
implemented.

6. Notes

Original issue reported on code.google.com by jeff.m...@continuent.com on 9 Sep 2014 at 6:41

[GoogleCodeExporter]

This issue was updated by revision r2546.

Create a branch to work on EBS snapshot backup functionality

Original comment by jeffm...@gmail.com on 9 Sep 2014 at 6:58

[GoogleCodeExporter]

This issue was updated by revision r2573.

Initial commit of functionality to support file-copy snapshots and EBS 
snapshots through `trepctl backup` and `trepctl restore`

Original comment by jeffm...@gmail.com on 22 Sep 2014 at 6:21

[GoogleCodeExporter]

Original comment by jeff.m...@continuent.com on 22 Sep 2014 at 6:21

• Changed state: Started

[GoogleCodeExporter]

- In order to use this, the build must come from 
https://tungsten-replicator.googlecode.com/svn/branches/replicator-issue-999.
- Configure your instance with a dedicated volume for MySQL logs and data
- Create the EC2 instance with an IAM profile that allows API access or create 
an /etc/tungsten/ebs_snapshot.ini file.
$> echo "access-key-id=############\nsecret-access-key=###############" > 
/etc/tungsten/ebs_snapshot.ini
- Configure the replicator to use the ebs-snapshot method
tpm configure servce --repl-backup-method=ebs-snapshot

At this point `trepctl backup` and `trepctl restore` should work normally. Once 
you hit the retention limit, the EC2 region will already have one extra 
snapshot present due to how snapshots are cleaned up.

Original comment by jeff.m...@continuent.com on 22 Sep 2014 at 7:00

[GoogleCodeExporter]

Instructions for setting up an additional EBS volume using Vagrant.

Update Vagrantfile to include this:

# Disable the requiretty setting and install puppet
                                aws.user_data = "#!/bin/bash\nhostname #{name.to_s()}\nperl -pi -e 's/^Defaults[ ]*requiretty/#Defaults requiretty/' /etc/sudoers\nyum install -y ruby18 rubygems18 rubygem18-aws-sdk ruby18-devel\nalternatives --set ruby /usr/bin/ruby1.8\nyum install -y puppet\nmkfs -t ext4 /dev/xvdb#{timer}"

                                aws.block_device_mapping = [
                                        {
                                                'DeviceName' => "/dev/sda1",
                                                'Ebs.VolumeSize' => 8,
                                                'Ebs.DeleteOnTermination' => true
                                        },
                                        {
                                                'DeviceName' => "/dev/sdb",
                                                'Ebs.VolumeSize' => 12,
                                                'Ebs.DeleteOnTermination' => true,
                                                'Ebs.VolumeType' => 'io1',
                                                'Ebs.Iops' => 300
                                        }
                                ]

Add this to your manifests/default.pp

file { "mysql-dir":
  ensure => directory,
  path => "/var/lib/mysql"
} ->
mount { "/var/lib/mysql":
  atboot => true,
  ensure => mounted,
  device => "/dev/xvdb",
  options => "defaults",
  fstype => "ext4"
} ->

Original comment by jeff.m...@continuent.com on 22 Sep 2014 at 7:09

[GoogleCodeExporter]

This issue was updated by revision r2577.

svn merge -r 2572:2573 
https://tungsten-replicator.googlecode.com/svn/branches/replicator-issue-999 .
Initial commit of functionality to support file-copy snapshots and EBS 
snapshots through `trepctl backup` and `trepctl restore`

Original comment by jeffm...@gmail.com on 23 Sep 2014 at 7:44

[GoogleCodeExporter]

Original comment by jeff.m...@continuent.com on 13 Oct 2014 at 6:18

• Added labels: FixedIn-3.0.1
• Removed labels: FixedIn-3.1.0

[GoogleCodeExporter]

Original comment by mc.br...@continuent.com on 19 Dec 2014 at 3:34

• Added labels: FixedIn-3.1.0
• Removed labels: FixedIn-3.0.1

[GoogleCodeExporter]

Original comment by linas.vi...@continuent.com on 19 Jan 2015 at 2:20

• Added labels: FixedIn-4.0.1
• Removed labels: FixedIn-3.1.0