Regression in 3.0.356.0: S3 access ignores HTTP proxy settings

[dpryden]

In our environment, HTTP/HTTPS access requires a proxy. We have configured proxy settings following the documented instructions at Configure SSM Agent to use a proxy, which sets the http_proxy, https_proxy, and no_proxy environment variables.

We have found that, since upgrading SSM Agent to 3.0.356.0, the agent is now unable to upload logs to S3. Commands with a configured S3 bucket are taking a long time to execute, and the log files are never being uploaded. In the /var/log/amazon/ssm/errors.log file, we see errors like:

2020-11-20 17:07:09 ERROR [func1 @ context.go.274] failed uploading /var/lib/amazon/ssm/i-0779f4d5976e5e04e/document/orchestration/1a7618ac-1acd-4e21-b86d-4f66461346a5/awsupdateSsmAgent/stdout to s3://redacted-bucket/1a7618ac-1acd-4e21-b86d-4f66461346a5/i-0779f4d5976e5e04e/awsupdateSsmAgent/stdout
 err:RequestError: send request failed
caused by: Put https://redacted-bucket.s3.amazonaws.com/1a7618ac-1acd-4e21-b86d-4f66461346a5/i-0779f4d5976e5e04e/awsupdateSsmAgent/stdout: dial tcp 52.217.83.124:443: connect: connection timed out

(Edit: I pasted the wrong log message at first, corrected.)

It appears that this regression was introduced by commit 7076f7fe89304002134c2289b2b49d4a14b426ec.

Specifically, I believe the issue is that newS3BucketRegionHeaderCapturingTransport (in agent/s3util/crossregion.go line 385) constructs a new http.Transport, bypassing any default configuration. Instead, I think it should use http.DefaultTransport, or else something like &http.Transport{Proxy: http.ProxyFromEnvironment}.

[danr-amz]

Thank you for bringing this to our attention. We are actively working on a fix.

[dpryden]

I see commit a962512229cde0bfbc3fcce1bcb4ad67a028bfe6 was recently merged, and release 3.0.431.0 was just created. Do I understand correctly that this release should fix this issue?

[danr-amz]

@dpryden Yes, this issue should be fixed in release 3.0.431.0.