wamdam / backy2

backy2: Deduplicating block based backup software for ceph/rbd, image files and devices
http://backy2.com/
Other
194 stars 39 forks source link

How to backup to s3? #24

Closed DeanKamali closed 5 years ago

DeanKamali commented 5 years ago

I have commented out type: backy2.data_backends.fileand only backy2.data_backends.s3 remains, yet, when I run backup

lvcreate --size 1G --snapshot --name snap /dev/ubuntu-vg/data
backy2 backup file:///dev/ubuntu-vg/snap root
lvremove -y /dev/ubuntu-vg/snap

Backup data gets stored on the host and I'm not seeing any backup data on my S3 bucket.

Here is my config

# backy configuration for backy2 Version 2

[DEFAULTS]
# Where should the backy logfile be placed?
# Backy will by default log INFO, WARNING and ERROR to this log.
# If you also need DEBUG information, please start backy with '-v'.
logfile: /var/log/backy.log

# Default block size. 4MB are recommended.
# DO NOT CHANGE WHEN BACKUPS EXIST!
block_size: 4194304

# Hash function to use. Use a large one to avoid collisions.
# DO NOT CHANGE WHEN BACKUPS EXIST!
hash_function: sha512

# for some operations, full backy or single versions need to be locked for
# simulatanous access. In the lock_dir we will create backy_*.lock files.
lock_dir: /run

# To be able to find other backys running, we need a system-wide unique name
# for all backy processes.
# DO NOT CHANGE WHILE backy2 PROCESSES ARE RUNNING!
process_name: backy2

# Allow rm of backup versions after n days (set to 0 to disable, i.e. to be
# able to delete any version)
disallow_rm_when_younger_than_days: 6

[MetaBackend]
# Of which type is the Metadata Backend Engine?
# Available types:
#   backy2.meta_backends.sql

#######################################
# backy2.meta_backends.sql
#######################################
type: backy2.meta_backends.sql

# Which SQL Server?
# Available servers:
#   sqlite:////path/to/sqlitefile
#   postgresql:///database
#   postgresql://user:password@host:port/database
engine: sqlite:////var/lib/backy2/backy.sqlite

[DataBackend]
# Which data backend to use?
# Available types:
#   backy2.data_backends.file
    backy2.data_backends.s3

#######################################
# backy2.data_backends.file
#######################################
#type: backy2.data_backends.file

# Store data to this path. A structure of 2 folders depth will be created
# in this path (e.g. '0a/33'). Blocks of DEFAULTS.block_size will be stored
# there. This is your backup storage!
#path: /var/lib/backy2/data

# How many writes to perform in parallel. This is useful if your backup space
# can perform parallel writes faster than serial ones.
#simultaneous_writes: 5

# How many reads to perform in parallel. This is useful if your backup space
# can perform parallel reads faster than serial ones.
#simultaneous_reads: 5

# Bandwidth throttling (set to 0 to disable, i.e. use full bandwidth)
# bytes per second
#bandwidth_read: 78643200
#bandwidth_write: 78643200

#######################################
# backy2.data_backends.s3
#######################################
type: backy2.data_backends.s3

# Your s3 access key
aws_access_key_id: JGMSOIKRWE4MHXM7KIRR

# Your s3 secret access key
aws_secret_access_key: 6DFft6LEu2H9AkWpgI3al4PZn8V6auDa9_MAwkAe

# Your aws host (IP or name)
host: 172.18.18.40

# The port to connect to (usually 80 if not secure or 443 if secure)
port: 9000

# Use HTTPS?
is_secure: false

# Store to this bucket name:
bucket_name: backy2

# How many s3 puts to perform in parallel
simultaneous_writes: 5

# How many reads to perform in parallel. This is useful if your backup space
# can perform parallel reads faster than serial ones.
simultaneous_reads: 5

# Bandwidth throttling (set to 0 to disable, i.e. use full bandwidth)
# bytes per second
bandwidth_read: 78643200
bandwidth_write: 78643200

[NBD]
# Where to cache backy blocks when NBD Server is used
# Enterprise version only
cachedir: /tmp

[io_file]
# Configure the file IO (file://<path>)
# This is for a file or a blockdevice (e.g. /dev/sda)

# How many parallel reads are permitted? (also affects the queue length)
simultaneous_reads: 5

[io_rbd]
# Configure the rbd IO (rbd://<pool>/<imagename>[@<snapshotname>])
# This accepts rbd images in the form rbd://pool/image@snapshot or rbd://pool/image

# Where to look for the ceph configfile to read keys and hosts from it
ceph_conffile: /etc/ceph/ceph.conf

# How many parallel reads are permitted? (also affects the queue length)
simultaneous_reads: 10

# When restoring images, new images are created (if you don't --force). For these
# newly created images, use these features:
new_image_features:
    RBD_FEATURE_LAYERING
    RBD_FEATURE_EXCLUSIVE_LOCK
#RBD_FEATURE_STRIPINGV2
#RBD_FEATURE_OBJECT_MAP
#RBD_FEATURE_FAST_DIFF
#RBD_FEATURE_DEEP_FLATTEN
wamdam commented 5 years ago

Looks ok on the first view. Maybe it's not the config that's in use? backy2 looks at these paths for the config: /etc/backy.cfg /etc/backy/backy.cfg /etc/backy/conf.d/* ~/.backy.cfg ./backy.cfg

Double check if there's no other config around.