search

aws / amazon-ssm-agent

An agent to enable remote management of your EC2 instances, on-premises servers, or virtual machines (VMs).
https://aws.amazon.com/systems-manager/
Apache License 2.0
1.04k stars 323 forks source link

SSM Port forwarding session doesn't check if the remote port is alive #367

Closed SuchismitaGoswami closed 3 years ago

SuchismitaGoswami commented 3 years ago

Operating System :

NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

Issue

AWS Region : us-east-2

I tried to create a port forwarding session to a service running on an ec2 (private-subnet) instance. It worked as expected if the remote service is accepting the connection. However, if the remote port is not accessible/alive, the ssm portforwarding session does not through error. It log the message as below and wait for connection request - 

Starting session with SessionId: XXXXXXXXXXXXXXXXXXXXXXXXXX
Port 8999 opened for sessionId XXXXXXXXXXXXXXXXXXXXXXXXXX.

Furthermore, on sending request to the local bindaddress: bindport, it fails and log the message as below

Connection accepted for session XXXXXXXXXXXXXXXXXXXXXXXXX.                                                                                                                                    Cannot perform start session: write tcp IP:38362->52.95.17.95:443: write: broken pipe

I have verified that on Windows, both the cases worked absolutely fine!>

YujiaozhAws commented 3 years ago

Thanks for reaching out. We'll investigate this.

nitikaaws commented 3 years ago

We looked into this further and unable to replicate the issue. After testing on Ubuntu, AmazonLinux2 and Windows2019, we found behavior to be consistent. On connecting to lbindaddress:bindport, below messages were displayed.

Connection accepted for session [SESSION_ID]
Connection to destination port failed, check SSM Agent logs.

Could you please confirm if this is the behavior you saw on Windows? If so, could you please check SSM Agent version on Ubuntu and upgrade if it is lower than the agent version installed on Windows? In case issue still persists, please provide agent logs from ubuntu instance for problematic session. This will help us troubleshoot further. Thanks!

SuchismitaGoswami commented 3 years ago

Thanks a lot for your quick support. I will reproduce the behavior and do accordingly as you suggested. I'll let you know if I face the same issue again.

nitikaaws commented 3 years ago

Please reach out to us if you face this issue again.