tbarbugli / cassandra_snapshotter

A tool to backup cassandra nodes using snapshots and incremental backups on S3
Other
222 stars 122 forks source link

AccessDenied error with encryption #87

Open toddrosner opened 8 years ago

toddrosner commented 8 years ago

I've just setup the snapshotter, and when I run the command, it compresses and seemingly transfers everything to S3 (looks like it's all there), but then it errors at the end with the nodetool ring command as follows:

[hostname] run: /usr/bin/nodetool ring
Traceback (most recent call last):
  File "/usr/local/bin/cassandra-snapshotter", line 9, in <module>
    load_entry_point('cassandra-snapshotter==0.5.0', 'console_scripts', 'cassandra-snapshotter')()
  File "/usr/local/lib/python2.7/site-packages/cassandra_snapshotter/main.py", line 280, in main
    run_backup(args)
  File "/usr/local/lib/python2.7/site-packages/cassandra_snapshotter/main.py", line 76, in run_backup
    worker.snapshot(snapshot)
  File "/usr/local/lib/python2.7/site-packages/cassandra_snapshotter/snapshotting.py", line 351, in snapshot
    self.write_ring_description(snapshot)
  File "/usr/local/lib/python2.7/site-packages/cassandra_snapshotter/snapshotting.py", line 400, in write_ring_description
    self.write_on_S3(snapshot.s3_bucket, ring_path, content)
  File "/usr/local/lib/python2.7/site-packages/cassandra_snapshotter/snapshotting.py", line 394, in write_on_S3
    key.set_contents_from_string(content)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1426, in set_contents_from_string
    encrypt_key=encrypt_key)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 750, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 951, in _send_file_internal
    query_args=query_args
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 665, in make_request
    retry_handler=retry_handler
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 940, in _mexe
    request.body, request.headers)
  File "/usr/lib/python2.7/dist-packages/boto/s3/key.py", line 884, in sender
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6C23CB43B06B5909</RequestId><HostId>z6yqnOVBCfLjnF0xTB1jyZaKpsVd4Q+Vgizvl9iJCxbgH573MFXuu9UCfCKlv1nvn2Fp/Ronxlo=</HostId></Error>

I've just discovered that this is happening because of encryption. Without the --s3-ssenc flag and with the bucket encryption policy removed, everything completes. Any known reason why the nodetool ring fails when encryption is enabled?

S3 bucket policy

{
    "Version": "2012-10-17",
    "Id": "PutObjPolicy",
    "Statement": [
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::s3-bucket-name/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        }
    ]
}
tbarbugli commented 8 years ago

what version of the tool are you using?

toddrosner commented 8 years ago

The version is 1.0.0