search

tbarbugli / cassandra_snapshotter

A tool to backup cassandra nodes using snapshots and incremental backups on S3
Other
222 stars 122 forks source link

Add ability to specify S3 bucket region #17

Closed bart613 closed 10 years ago

bart613 commented 10 years ago

These two commits add ability to specify S3 bucket region.

If you're using eg. bucket in eu-west-1 and uploading large-ish files, quite often you get socket.error: [Errno 104] Connection reset by peer errors due to the traffic being routed through US (us-east-1) which is the default S3 endpoint. This was happening quite often to me.

$ cassandra-snapshotter --aws-access-key-id ACCESSKEY --aws-secret-access-key SECRETKEY --s3-bucket-name aws-s3-bucket --s3-base-path staging backup --hosts=node-a,node-b,node-c --user=bart
[node-b] Executing task 'node_start_backup'
[node-c] Executing task 'node_start_backup'
[node-a] Executing task 'node_start_backup'
[node-a] sudo: /usr/bin/nodetool snapshot -t 20140515114434
[node-c] sudo: /usr/bin/nodetool snapshot -t 20140515114434
[node-b] sudo: /usr/bin/nodetool snapshot -t 20140515114434
[node-a] out: Requested creating snapshot for: all keyspaces
[node-a] out: Snapshot directory: 20140515114434
[node-a] out:

[node-b] out: Requested creating snapshot for: all keyspaces
[node-b] out: Snapshot directory: 20140515114434
[node-b] out:

[node-c] out: Requested creating snapshot for: all keyspaces
[node-c] out: Snapshot directory: 20140515114434
[node-c] out:

[node-b] Executing task 'upload_node_backups'
[node-c] Executing task 'upload_node_backups'
[node-a] Executing task 'upload_node_backups'
[node-a] sudo: python -c "import os; print os.path.join(*['/var/lib/cassandra/data/', '*', '*', 'snapshots', '20140515114434', '*'])"
[node-c] sudo: python -c "import os; print os.path.join(*['/var/lib/cassandra/data/', '*', '*', 'snapshots', '20140515114434', '*'])"
[node-b] sudo: python -c "import os; print os.path.join(*['/var/lib/cassandra/data/', '*', '*', 'snapshots', '20140515114434', '*'])"
[node-b] sudo: python -c "import glob; print '\n'.join(glob.glob('/var/lib/cassandra/data/*/*/snapshots/20140515114434/*'))"
[node-a] sudo: python -c "import glob; print '\n'.join(glob.glob('/var/lib/cassandra/data/*/*/snapshots/20140515114434/*'))"
[node-c] sudo: python -c "import glob; print '\n'.join(glob.glob('/var/lib/cassandra/data/*/*/snapshots/20140515114434/*'))"
[node-c] put: /var/folders/dn/b5h8kyhn1f5dk0d7_bv3lnmr0000gn/T/tmpR4Hr9_ -> /tmp/tmpR4Hr9_
[node-b] put: /var/folders/dn/b5h8kyhn1f5dk0d7_bv3lnmr0000gn/T/tmpbC6P8m -> /tmp/tmpbC6P8m
[node-a] put: /var/folders/dn/b5h8kyhn1f5dk0d7_bv3lnmr0000gn/T/tmpmwr93Z -> /tmp/tmpmwr93Z
[node-c] sudo: cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-c  put --manifest=/tmp/tmpR4Hr9_ --concurrency=4
[node-b] sudo: cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-b  put --manifest=/tmp/tmpbC6P8m --concurrency=4
[node-a] sudo: cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-a  put --manifest=/tmp/tmpmwr93Z --concurrency=4
[node-a] out: lzop 1.02rc1
[node-a] out: LZO library 2.03
[node-a] out: Copyright (C) 1996-2005 Markus Franz Xaver Johannes Oberhumer
[node-c] out: lzop 1.02rc1
[node-c] out: LZO library 2.03
[node-c] out: Copyright (C) 1996-2005 Markus Franz Xaver Johannes Oberhumer
[node-b] out: lzop 1.02rc1
[node-b] out: LZO library 2.03
[node-b] out: Copyright (C) 1996-2005 Markus Franz Xaver Johannes Oberhumer
[node-c] out: Traceback (most recent call last):
[node-c] out:   File "/usr/local/bin/cassandra-snapshotter-agent", line 9, in <module>
[node-c] out:     load_entry_point('cassandra-snapshotter==0.2.2', 'console_scripts', 'cassandra-snapshotter-agent')()
[node-c] out:   File "/usr/local/lib/python2.6/dist-packages/cassandra_snapshotter/agent.py", line 111, in main
[node-c] out:     args.concurrency
[node-c] out:   File "/usr/local/lib/python2.6/dist-packages/cassandra_snapshotter/agent.py", line 79, in put_from_manifest
[node-c] out:     for _ in pool.imap(upload_file, ((bucket, f, destination_path(s3_base_path, f)) for f in files)):
[node-c] out:   File "/usr/lib/python2.6/multiprocessing/pool.py", line 520, in next
[node-c] out:     raise value
[node-c] out: socket.error: [Errno 104] Connection reset by peer
[node-c] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-c  put --manifest=/tmp/tmpR4Hr9_ --concurrency=4
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-c  put --manifest=/tmp/tmpR4Hr9_ --concurrency=4"

Aborting.
[node-b] out: Traceback (most recent call last):
[node-b] out:   File "/usr/local/bin/cassandra-snapshotter-agent", line 9, in <module>
[node-b] out:     load_entry_point('cassandra-snapshotter==0.2.2', 'console_scripts', 'cassandra-snapshotter-agent')()
[node-b] out:   File "/usr/local/lib/python2.6/dist-packages/cassandra_snapshotter/agent.py", line 111, in main
[node-b] out:     args.concurrency
[node-b] out:   File "/usr/local/lib/python2.6/dist-packages/cassandra_snapshotter/agent.py", line 79, in put_from_manifest
[node-b] out:     for _ in pool.imap(upload_file, ((bucket, f, destination_path(s3_base_path, f)) for f in files)):
[node-b] out:   File "/usr/lib/python2.6/multiprocessing/pool.py", line 520, in next
[node-b] out:     raise value
[node-b] out: socket.error: [Errno 104] Connection reset by peer
[node-b] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-b  put --manifest=/tmp/tmpbC6P8m --concurrency=4
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-b  put --manifest=/tmp/tmpbC6P8m --concurrency=4"

Aborting.
[node-a] out: Traceback (most recent call last):
[node-a] out:   File "/usr/local/bin/cassandra-snapshotter-agent", line 9, in <module>
[node-a] out:     load_entry_point('cassandra-snapshotter==0.2.2', 'console_scripts', 'cassandra-snapshotter-agent')()
[node-a] out:   File "/usr/local/lib/python2.6/dist-packages/cassandra_snapshotter/agent.py", line 111, in main
[node-a] out:     args.concurrency
[node-a] out:   File "/usr/local/lib/python2.6/dist-packages/cassandra_snapshotter/agent.py", line 79, in put_from_manifest
[node-a] out:     for _ in pool.imap(upload_file, ((bucket, f, destination_path(s3_base_path, f)) for f in files)):
[node-a] out:   File "/usr/lib/python2.6/multiprocessing/pool.py", line 520, in next
[node-a] out:     raise value
[node-a] out: socket.error: [Errno 104] Connection reset by peer
[node-a] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-a  put --manifest=/tmp/tmpmwr93Z --concurrency=4
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "cassandra-snapshotter-agent --aws-access-key-id=ACCESSKEY --aws-secret-access-key=SECRETKEY --s3-bucket-name=aws-s3-bucket --s3-base-path=staging/20140515114434/node-a  put --manifest=/tmp/tmpmwr93Z --concurrency=4"

Aborting.

Fatal error: One or more hosts failed while executing task 'upload_node_backups'

Aborting.
[node-b] Executing task 'clear_node_snapshot'
[node-c] Executing task 'clear_node_snapshot'
[node-a] Executing task 'clear_node_snapshot'
[node-a] sudo: /usr/bin/nodetool clearsnapshot -t "20140515114434"
[node-c] sudo: /usr/bin/nodetool clearsnapshot -t "20140515114434"
[node-b] sudo: /usr/bin/nodetool clearsnapshot -t "20140515114434"
[node-a] out: Requested clearing snapshot for: all keyspaces
[node-a] out:

[node-b] out: Requested clearing snapshot for: all keyspaces
[node-b] out:

[node-c] out: Requested clearing snapshot for: all keyspaces
[node-c] out:

Patches in this PR add ability to specify S3 bucket region with fallback to it being us-east-1 by default. This speeds up transfer speeds as well.

tbarbugli commented 10 years ago

Hi, Thank You for your contribution! Your changes are merged on master branch; I am going to give them a good test and then release a new version of the tool. Dont forget to add yourself to the CONTRIBUTORS file ;)

Cheers, Tommaso