search

aws / amazon-ssm-agent

An agent to enable remote management of your EC2 instances, on-premises servers, or virtual machines (VMs).
https://aws.amazon.com/systems-manager/
Apache License 2.0
1.05k stars 323 forks source link

Session logs aren't being delivered to s3 bucket #311

Closed yellowmamba closed 3 years ago

yellowmamba commented 3 years ago

When I check the ssm error logs var/log/amazon/ssm/errors.log, I can see uploading to s3 has failed 

020-10-09 10:27:19 ERROR [S3Upload @ s3util.go.108] [ssm-session-worker] [42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0] [DataBackend] [pluginName=Standard_Stream] Attempt %!s(int=4): Failed uploading /var/lib/amazon/ssm/<instance_id>/session/orchestration/42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0/Standard_Stream/42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0.log to s3://<bucket_name>/ssm/<aws_account_id>/42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0.log err:AccessDenied: Access Denied

However, if I manually run aws s3 cp command with --sse (my bucket is encrypted with the default AES256), the upload succeeded.

aws s3 cp /var/lib/amazon/ssm/<instance_id>/session/orchestration/42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0/Standard_Stream/42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0.log s3://<bucket_name>/ssm/<aws_account_id>/42ba7ac1-6122-4f3a-aa64-5f18f81eff1d-0959abc60267189e0.log --sse

Can anyone point out where I might have got things wrong?

The agent version is 3.0.222.0

ranjrish commented 3 years ago

This is due to the fact that the sse header is missing during s3 upload, https://github.com/aws/amazon-ssm-agent/blob/0aa412ef856ffc58e9cb0e5e8b08b77af48aef62/agent/s3util/s3util.go#L93-L99

Working on adding the 'AES256' and 'KMS' SSE headers

yellowmamba commented 3 years ago

@ranjrish Yes I did notice this code and I was going to submit a PR. However, we do have another aws account in which ssm logs are able to be uploaded to an AES256-enabled bucket, under the same version of ssm agent. So I wasn't sure whether this was a code issue as you highlighted.

hpapadopoulos commented 3 years ago

We are seeing the same issue on some of the accounts. The problem started when we enabled an SCP policy to request sse headers on the source account that the ssm agent was running (we have different account where the logs are stored).

Bucket with the exact same configuration and on the same account as the one not working, can still receive logs from an account where the SCP isn't enabled.

ranjrish commented 3 years ago

This happens when an SCP which ensures only encrypted files are being uploaded to the bucket. We are working on a fix for this

yellowmamba commented 3 years ago

In our case, we don't seem to have any suspicious SCP applied.

yellowmamba commented 3 years ago

@ranjrish Correcting my comment above, we do have an SCP that denies s3 access if no SSE header is present, and the account in which things are working does not have the same SCP applied. Please let me know when this issue has been resolved.

ranjrish commented 3 years ago

This issue has been resolved in v3.0.529.0