Consul snapshot agent backup file to aws s3 error

[lcgkm]

Currently, we setup a consul snapshot agent to backup our consul data. But we found following error message:

Jul 10 16:35:25 ip-10-211-178-16 consul[4284]:             Lock Key: "consul-snapshot/lock"
Jul 10 16:35:25 ip-10-211-178-16 consul[4284]:         Max Failures: 3
Jul 10 16:35:25 ip-10-211-178-16 consul[4284]:     Snapshot Storage: Amazon S3 -> Region: "xx-xxxxx-x" Bucket: "security-vault-backup" Key Prefix: "consul
Jul 10 16:35:25 ip-10-211-178-16 consul[4284]: ==> Log data will now stream in as it occurs:
Jul 10 16:35:25 ip-10-211-178-16 consul[4284]:     2018/07/10 16:35:25 [INFO] Waiting to obtain leadership...
Jul 10 16:35:25 ip-10-211-178-16 consul[4284]:     2018/07/10 16:35:25 [INFO] Obtained leadership
Jul 10 16:35:25 ip-10-211-178-16 consul[4284]:     2018/07/10 16:35:25 [DEBUG] Taking a snapshot...
Jul 10 16:35:26 ip-10-211-178-16 consul[4284]:     2018/07/10 16:35:26 [INFO] Saved snapshot with id 1531208125988786033
Jul 10 16:35:26 ip-10-211-178-16 consul[4284]:     2018/07/10 16:35:26 [ERR] Snapshot failed (will retry at next interval): AccessDenied: Access Denied
Jul 10 16:35:26 ip-10-211-178-16 consul[4284]:         status code: 403, request id: xxxxxxxxxx

Our target S3 Bucket Permissions is:

s3:ListBucket on arn:aws:s3:::mybucket
s3:GetObject on arn:aws:s3:::mybucket/path/to/my/key
s3:PutObject on arn:aws:s3:::mybucket/path/to/my/key

But we can see the snapshot files have uploaded to aws s3. Why I still get error message?

Maybe our target S3 Bucket Permissions not enough? Should we assign some other s3 permissions to our target S3 Bucket?

[pearkes]

Can you provide your configuration and if possible IAM policy? If you're using KMS or some of the other options it may affect the permissions required. It does require put, get, list, and delete on the specified key prefix.

[lcgkm]

My IAM policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "s3:ListObjects"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
            ],
            "Resource": [
                "arn:aws:s3:::consul-backup/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::consul-backup"
            ]
        }
    ]
}

If you're using KMS or some of the other options it may affect the permissions required.

No, I don't use KMS. But what other options for consul snapshot? Can you share some details for this?

[lcgkm]

I lost "s3:ListBucketVersions" https://www.consul.io/docs/commands/snapshot/agent.html

{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::consul-data/consul-snapshots/consul-*.snap" }, { "Sid": "", "Effect": "Allow", "Action": [ "s3:ListBucketVersions", "s3:ListBucket" ], "Resource": "arn:aws:s3:::consul-data" } ] }

[Leectan]

@lcgkm, did you figure out what was the issue that is causing this? I'm sure folks who gone through the same issue would like to know as well.

[lcgkm]

@lcgkm, did you figure out what was the issue that is causing this? I'm sure folks who gone through the same issue would like to know as well.

I lost "s3:ListBucketVersions" Please check the example IAM policy document from: https://www.consul.io/docs/commands/snapshot/agent.html

[jmariondev]

For others that land here via Google, I had the same issue but for me it was caused by using a KMS key for S3 encryption. The relevant IAM policy addition was this:

        {
            "Effect": "Allow",
            "Action": [
                "kms:GenerateDataKey"
            ],
            "Resource": [
                "arn:aws:kms:<region>:<key_identifier>"
            ]
        }

Since I had snapshot rotation disabled, the only other permission I needed was s3:PutObject.

[pearkes]

Thanks @jmariondev we should probably add a note about KMS to the docs given it may happen again.