search

aws / session-manager-plugin

This plugin helps you to use the AWS Command Line Interface (AWS CLI) to start and end sessions to your managed instances
Apache License 2.0
261 stars 70 forks source link

Cannot start session from sdk with version 1.2.688.0 #101

Open yongzhang opened 1 day ago

yongzhang commented 1 day ago

Hi, team

We're using golang sdk to start ssm session, but it is broken after session-manager-plugin upgraded to 1.2.688.0, I got error:

2024-11-13 18:41:45 ERROR [getV4SignatureHeader @ websocketchannel.go.138] Failed to sign websocket, NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2024-11-13 18:41:45 ERROR [Open @ websocketchannel.go.199] Failed to get the v4 signature, NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

verbose message:

2024-11-14 17:54:50 ERROR [getV4SignatureHeader @ websocketchannel.go.141] Failed to sign websocket, NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Seems like it is caused by this change.

And I'm afraid this line broken it: https://github.com/aws/session-manager-plugin/blob/80869bb97a01708e01ed4ff05fe4643c66759404/src/sessionmanagerplugin/session/sessionhandler.go#L41

I do not reply on local config or env to get credentials in my code, in my case, I created aws config like this with aws-go-sdk-v2:

    cfg, err := config.LoadDefaultConfig(ctx,
        config.WithRegion(resp.GetRegion()),
        config.WithCredentialsProvider(credentials.StaticCredentialsProvider{
            Value: aws.Credentials{
                AccessKeyID:     resp.GetAccessKeyId(),
                SecretAccessKey: resp.GetSecretAccessKey(),
                SessionToken:    resp.GetSessionToken(),
            },
        }),
    )

...
StartSession(ssm.NewFromConfig(cfg) ...
hatchetaustralia commented 15 hours ago

Not sure if directly related to the same issue as you, but we saw a breaking change for us between 1.2.677.0 and 1.2.688.0. Specifically, when establishing an SSH connection via the data channel over SSM, the handshake would fail using the latest plugin. Logs from the target machines running SSM pointed us to the handshake failure.

Rolling back to the previous version 1.2.677.0 resolved the issue.

2024-11-15 03:58:11.3022 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Setting up datachannel for session: botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy, requestId: c14233a5-2e23-42b1-b09b-7182694300c5, clientId:
2024-11-15 03:58:11.3609 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Opening websocket connection to: wss://ssmmessages.ap-southeast-2.amazonaws.com/v1/data-channel/botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy?role=publish_subscribe
2024-11-15 03:58:11.4023 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Successfully opened websocket connection to: 99.83.82.120:443
2024-11-15 03:58:11.4024 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Starting websocket pinger
2024-11-15 03:58:11.4025 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Initiating Handshake
2024-11-15 03:58:11.4026 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Starting websocket listener
2024-11-15 03:58:26.4037 ERROR [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Encountered error while initiating handshake. Handshake timed out. Please ensure that you have the latest version of the session manager plugin.
2024-11-15 03:58:28.4041 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Closing datachannel with channel Id botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy
2024-11-15 03:58:28.4084 INFO [ssm-agent-worker] [MessageService] [EngineProcessor] [BasicExecuter] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] requested terminate messaging worker, destroying the channel
2024-11-15 03:58:28.4041 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Closing websocket channel connection to: wss://ssmmessages.ap-southeast-2.amazonaws.com/v1/data-channel/botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy?role=publish_subscribe
2024-11-15 03:58:28.4043 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Ending websocket pinger
2024-11-15 03:58:28.4044 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Ending the channel listening routine since the channel is closed
2024-11-15 03:58:28.4044 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Ending websocket listener
2024-11-15 03:58:28.4044 INFO [ssm-session-worker] [botocore-session-1731643091-xtr7trt348ebhfaxf8bfkek3gy] [DataBackend] [pluginName=Port] Successfully closed websocket connection to: 99.83.82.120:443