search

aws / amazon-ssm-agent

An agent to enable remote management of your EC2 instances, on-premises servers, or virtual machines (VMs).
https://aws.amazon.com/systems-manager/
Apache License 2.0
1.05k stars 323 forks source link

Port forwarding freeze (AWS-StartPortForwardingSession) #294

Closed m1keil closed 3 years ago

m1keil commented 4 years ago

Hello,

I'm using Ubuntu 16.04 (eu-west-1 ami-1f83aa79) with SSM agent manually installed from the deb package (v2.3.1509.0).

I'm testing out AWS-StartPortForwardingSession document and trying to forward ports to a "complex" application Web UI such as Hashicorp's consul.

In the network tab of the inspect console (Chrome) I can see that the first few requests are passing fine but at some point (about 5-6 HTTP requests in), the requests freeze and data stream stops. Any attempts to issue requests in parallel yield no response from the tunnel: image

Restarting the session helps to "unblock" but this happens again when the webpage tries to load the other assets.

I'm able to replicate this over multiple instances and different applications. No error in amazon-ssm-agent unit logs. This does not happen with a regular tunnel over SSH.

session-manager-plugin version 1.1.61.0.

m1keil commented 4 years ago

Update:

I'm able to reproduce the issue with Latest ubuntu 20.04 (eu-west-1 ami-0127d62154efde733) and the default amazon-ssm-manager snap that came pre-installed (2.3.978.0).

Seems like this functionality is able to push very small amount of data before getting into a deadlock which makes it very unreliable.

Session ID for example - michael@strigo.io-0183a8820663a8e20

Seems to be related: #221

nitikagoyal87 commented 4 years ago

Thanks for your feedback. We will look into this!

Menahem1 commented 4 years ago

Hello,

Are there any evolutions ?

Thanks

m1keil commented 3 years ago

After more digging, this seems to be the same issue as reported by #259 & #273. Here's a similar thread on the AWS support forums with the same response from AWS.

SSM will only allow a single connection. Any attempts to establish a second connection will block until the first connection is closed. It's not clear to me why in my case that second connection blocks forever and not resumes once the first connection is done. Might be keepalive, or some timeout along the route. Haven't dug past this point.

The only workaround I found for this is to avoid AWS-StartPortForwardingSession and use AWS-StartSSHSession with native ssh client port forwarding instead. This way the connection multiplexing will be done on your side.

It would be great if AWS can acknowledge this limitation and update documents to make this clear.

riteshchaman commented 3 years ago

AWS has released a new feature in SM which may solve this issue. Upgrade your ssm-agent to 3.0.222.0 or later and Session Manager plugin to 1.2 or later. https://aws.amazon.com/about-aws/whats-new/2020/10/port-forwarding-sessions-created-sessions-manager-support-multiple-simultaneous-connections/

m1keil commented 3 years ago

@riteshchaman thanks for the FYI. I will test it and see if it helps.

m1keil commented 3 years ago

Can confirm this does seem to fix the issues I was having.

Seem to work fine when:

❯ /usr/local/bin/session-manager-plugin --version
1.2.7.0

and 

$ snap list
Name              Version    Rev    Tracking          Publisher   Notes
amazon-ssm-agent  3.0.284.0  3204   latest/candidate  aws✓        classic

Going to close this issue now.