Support for specifying a host for port forwarding.

[magJ]

Issue #208

Description of changes:

This is a long requested feature. SSH has long supported forwarding ports from remote hosts, this brings similar functionality to SSM sessions.

Session document

Ideally someone at amazon could add a host parameter to the AWS-StartPortForwardingSession document, or create a new public version of the document that has the host parameter.

In lieu of this, we can create one ourselves.

Create a copy of the AWS-StartPortForwardingSession document, and add a parameter for host. For example:

{
  "schemaVersion": "1.0",
  "description": "Document to start port forwarding session over Session Manager",
  "sessionType": "Port",
  "parameters": {
    "portNumber": {
      "type": "String",
      "description": "(Optional) Port number of the server on the instance",
      "allowedPattern": "^([1-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$",
      "default": "80"
    },
    "host": {
      "type": "String",
      "description": "(Optional) Host to connect to, will default to the local target instance host",
      "default": ""
    },
    "localPortNumber": {
      "type": "String",
      "description": "(Optional) Port number on local machine to forward traffic to. An open port is chosen at run-time if not provided",
      "allowedPattern": "^([0-9]|[1-9][0-9]{1,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])$",
      "default": "0"
    }
  },
  "properties": {
    "portNumber": "{{ portNumber }}",
    "host": "{{ host }}",
    "type": "LocalPortForwarding",
    "localPortNumber": "{{ localPortNumber }}"
  }
}

# create the session document
aws ssm create-document \
    --content 'file://AWS-StartPortForwardingSessionWithHost.json' \
    --name 'Custom-AWS-StartPortForwardingSession' \
    --document-type "Session"

Testing

For example, accessing RDS via an instance running SSM-Agent.

# start a port forwarding session, using a remote host
aws ssm start-session \
    --document-name 'Custom-AWS-StartPortForwardingSession' \
    --parameters 'portNumber=[3306],host=[aaaaaaa.bbbbbb.ap-southeast-2.rds.amazonaws.com],localPortNumber=[3306]' \
    --target '<instance-id>'

# in another terminal we can now access the resource at the remote host via the tunnel
mysql --host=localhost --port=3306

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[uright]

I am looking forward for this to be merged. This way we don't need to manage PEM and solely depends on IAM for bastion host.

[ghost]

I'm looking forward to being a merged, too.

[wiredin]

Also excited for this to be merged.

[laur1s]

Any news about when will this be merged?

[yuting-fan]

Hi folks,

Thank you for the pull request on this feature. Session Manager service team is looking into this request next and will update in this thread with any progress.

Regards, Yuting

[followben]

Sorry folks - almost a month on so I'm going to give this a bump again. Can we please get this merged?

[ronkorving]

I can only guess how many hours of peoples' lives are wasted by the lack of quality-of-life features like this. Does this really take a month to review?

edit: Apologies for the cynicism, but with the lack of communication from AWS (who no doubt are working hard to make us happy), we don't know if there's any progress happening on this particular issue, or if it's going to be 2023. Expectation management could really improve, folks.

[valentyntymku]

I am looking forward to this being merged as well :)

[dbermuehler]

Any updates on this topic? It would really help us if this feature is finally implemented.

[chrismazanec]

compiled from source and tested, works as expected 👌🏾 can this be merged please?

[bedge]

I dislike the "+1, me too as much as anyone does, but Jesus this seems like a special case perverse torture from the masters. For the love of God, please, merge this. Or you can add +1 to the number of people that must now download build and distribute a custom version of the agent. The sheer volume of wasted manpower here is incalculable. No, wait, lets focus on calculating that. It's a better use of my time that replicating everyone else here is already doing. No bad feelings towards the maintainers, just frustrated waiting for this.

Forgot to mention, yes, it works perfectly.

[jon-nona]

PR open for 5 months? This is crazy. Please can this get merged?

[followben]

Sorry to do this but specifically tagging @Thor-Bjorgvinsson , @aguman-aws, @danr-amz.

This PR has been outstanding for more than 6 months and represents a critical feature for a lot of us keen to use ssm & iam in place of ssh to allow devs access to bastion hosts.

@yuting-fan said it was under review back in Sept, but we've still no transparency or update to the holdup, and you can see I'm not the only one that's frustrated. Can we please get it merged or at least give us a reason why it can't be?

[nitikaaws]

Hi Everyone,

Session Manager team is actively working on this pull request and currently undergoing a detailed review process. We will keep this thread updated as we make progress.

Thanks, Nitika

[william-salt-cko]

I actually raised this with our TAM in September, we had a meeting with a product manager on the SSM team. We asked what was blocking this being merged and released, and the response was that there release process takes months and they would potentially look at this in Q1 this year. When I asked specifically about the process and why a (seemingly) trivial change cant be reviewed and released sooner, i didn't get much of a response. I get that sometimes changes like this can have security implications, and there are a lot of other considerations, but i wasn't given any in depth reasoning or explanation.

To be honest, I have lost confidence in this product after this whole process and we are looking at other solutions. I would love it if someone from AWS, perhaps @nitikaaws would be able to go into detail as to the process and reason for such a delay? Perhaps even an AWS developer could help us understand some of the context here? Just open communication would help everyone involved.

It becomes really frustrating when something is opensource, but actually contributing to a project or getting feedback isn't possible.

[nitikaaws]

Sorry for any inconvenience caused by this delay. SSM agent changes goes through various internal reviews and testing to avoid impacting existing users. We are targeting to release this by 3/31. Your further patience is greatly appreciated.

Thanks, Nitika

[lukaskeller]

hi @nitikaaws :) is the merging of this PR still planned for 3/31? really looking forward to this improvement! Thanks!

[injeniero]

@nitikaaws It would be great to know if this is going to happen on Q1 or you are moving this for later. I need to setup access to the machines and I would love to be capable of using this feature.

[jrmy1]

Hello Everyone,

Port forwarding to remote hosts is currently being released and will be live by 3/31. We will provide another update when it is live.

Thanks, Jeremy

[ronkorving]

@jeremychangy That's fantastic news! Thanks for the update 👍

[cheld]

@jeremychangy so...today is the big day....I hope you have fantastic news?

[william-salt-cko]

Exciting! I have been waiting 9 months for this!

[jrmy1]

Hello everyone,

An internal review is blocking this and we are working hard to get this released. I will provide updates in the next two weeks.

I apologize for any inconveniences with this delay.

Thanks, Jeremy

[dhu5432]

So excited for this this!

[magJ]

Gonna close this, since this feature ended up being implemented independently with commit https://github.com/aws/amazon-ssm-agent/commit/c2f6dc2ec3f5133366f0adaa11b63e3922b76c28

[followben]

Sorry if I'm slow @magJ - it's unclear to me how we can use this latest change to create a port forwarding session (specifically to RDS) on an EC2 bastion using ssm. Could you elaborate?

Also, is this released yet and, if so, how do we update the ssm-agent on our target instance?

[magJ]

Sorry if I'm slow @magJ - it's unclear to me how we can use this latest change to create a port forwarding session (specifically to RDS) on an EC2 bastion using ssm. Could you elaborate?

Also, is this released yet and, if so, how do we update the ssm-agent on our target instance?

I haven't had a chance to test out the new changes, and I'm not sure if the new version has been included in base images yet.
That said, the new changes look to be pretty much the same as the changes I proposed, albeit with some additional restrictions on which remote hosts you can access.

If you are running the latest agent version, and use a session document like my example in the description, then I would imagine it should work.

I don't know if amazon have released or will release updates to the default AWS-StartPortForwardingSession document.
They could potentially release a new document like AWS-StartPortForwardingSessionWithHost or something.

[followben]

Apols @magJ - I thought you were from AWS :) Nice work on this patch - frustrating when your contributions get re-implemented, but I guess the final result is (slightly) more sophisticated. If only it didn't take 10 months to get to it. Credit where it's due though: this functionality is a major win for all concerned, so thanks for laying the groundwork!

I did some digging, and it appears the 3.1.1188.0 hasn't been uploaded for distribution yet:

% curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/VERSION
3.1.1080.0

I built & deployed the tag myself & can confirm: cloning the AWS-StartPortForwardingSession document and adding the host parameter works as you described.

Any plans to release an official document @jeremychangy?

[followben]

Well it was nice while it lasted. The ssm-agent I compiled and installed from source that was working 3 days ago with this exact same custom document and instruction now returns:

% aws ssm start-session --target my-instance-id \
--document-name 'Custom-AWS-StartPortForwardingSession' \
--parameters 'portNumber=[5432],host=[some-aurora-cluster.some-cluster-east-1.rds.amazonaws.com],localPortNumber=[5431]'

An error occurred (BadRequest) when calling the StartSession operation: Currently, port forwarding sessions to remote hosts are not supported.

Any insight as to what's gone on here @jeremychangy @nitikaaws?

[psoares-resilient]

Well it was nice while it lasted. The ssm-agent I compiled and installed from source that was working 3 days ago with this exact same custom document and instruction now returns:

Hi!

I just tried compiling the agent from the 3.1.1188.0 and it worked (using a modified document as suggested above). My steps:

git clone git@work-github.com:aws/amazon-ssm-agent.git
git checkout tags/3.1.1188.0 -b custom-ssm-agent
docker image prune -f 
docker build -t ssm-agent-build-image .
docker run -it --rm --name ssm-agent-build-container -v `pwd`:/amazon-ssm-agent ssm-agent-build-image make build-release

Then I went for some tea. The binaries were compiled and built into the bin folder. I picked the version that suited my ec2: bin/linux_amd64/amazon-ssm-agent.rpm and moved it into an S3 bucket so I could download it from the ec2 itself. From there I remoted into the ec2:

aws ssm start-session --target i-0123456789abcdef
sudo -s
systemctl status amazon-ssm-agent
curl https://s3.[REGION].amazonaws.com/[BUCKET]/aws-ssm/amazon-ssm-agent.rpm -o "session-manager-plugin.rpm"
yum install -y ./session-manager-plugin.rpm
systemctl restart amazon-ssm-agent

I've checked the version I had running and the one I just installed and they were different. After restarting, I could connect to the remote host just fine.

aws ssm start-session --target i-0123456789abcdef --document-name SN-RemoteTunnel --parameters '{"portNumber":["443"],"localPortNumber":["9200"], "host":["vpc-opensearch-cluster-id.region.es.amazonaws.com"]}'

Where SN-RemoteTunnel is a clone of AWS-StartPortForwardingSession with the host parameter added:

[chrismazanec]

I've installed a fresh EC2 instance, ssm agent with host port forwarding is working out of the box providing that you've got adjusted Custom-AWS-StartPortForwardingSession document in place. the latest version available currently is 3.1.1188.0 so compilation is no longer required 🎉

> curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/VERSION
3.1.1188.0

[jrmy1]

Hello everyone,

We are still working on an internal review to release an AWS official document for this feature. We will keep you informed when there is an update.

Thanks, Jeremy

[followben]

I reinstalled the latest ssm release on my AL2 instance by sudo yum reinstall https://s3.us-east-1.amazonaws.com/amazon-ssm-us-east-1/latest/linux_amd64/amazon-ssm-agent.rpm and confirmed it's 3.1.1188.0 and running ok. However when using the exact same document and invocation format as @psoares-resilient and @magJ above I still get:

% aws ssm start-session --target my-instance-id \
--document-name 'Custom-AWS-StartPortForwardingSession' \
--parameters '{"portNumber":["5432"],"localPortNumber":["5431"], "host":["my-aurora-cluster.cluster-identifier.us-east-1.rds.amazonaws.com"]}'

An error occurred (BadRequest) when calling the StartSession operation: Currently, port forwarding sessions to remote hosts are not supported.

Why would this now be enabled/ supported for some instances but has been switched off for others @jeremychangy? To confirm: port forwarding via the latest ssm-agent previously worked on this exact same instance and client. How can I work around this and reenable it?

[magJ]

An error occurred (BadRequest) when calling the StartSession operation: Currently, port forwarding sessions to remote hosts are not supported.

Given that I can't see any reference to that error message in the agent codebase, I assume it's coming from the SSM API backend itself.
Whereas previously SSM would just pass the document fields unmolested to the agent, it now seems like it's looking for the host field and blocking the request if found.

Seems very strange to me that it the functionality would be artificially limited like that.

I wonder if you compiled your own agent, and changed the host field name, to something else if the functionality would work.

[richard-wgp]

We are still working on an internal review to release an AWS official document for this feature. We will keep you informed when there is an update.

@jeremychangy I would guess that you're implementing something like this, rather than the allow anything for host in the document @magJ first posted, which is what I've been using in my custom document:

    "host": {
      "type": "String",
      "description": "(Optional) Host to connect to, will default to the local target instance host",
      "allowedPattern": "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$|^(([a-zA-Z]|[a-zA-Z][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z]|[A-Za-z][A-Za-z0-9\\-]*[A-Za-z0-9])$",
      "default": ""
    },

[bedge]

What changed to break this existing patched mechanism?

An amazon-ssm-agent built from the above patch that I've been distributing internally as my own 3.1.0.0-1 version, and the same ssm document that I've been using for the past several months that was working flawlessly, suddenly stops working. WTF?

Same exact instance/kernel/boot/etc that used to work

Sure, I'll repeat the same command as others have already posted for clarity rather than adding anything new:

[I] ➜ aws ssm start-session --target i-0488cc8977dXXXXXX --document-name MyXXX-StartPortForwardingSessionHost --parameters '{"portNumber":["1521"],"host":["db.xx...eu-central-1.rds.amazonaws.com"],"localPortNumber":["1521"]}'

An error occurred (BadRequest) when calling the StartSession operation: Currently, port forwarding sessions to remote hosts are not supported.

Is there some AWS ssm magic glue that's parsing document requests?

Can't be a new aws CLI either as it broke for a whole team of devs at the same time.

Anyone have a workaround for this current "worse than before you tried to fix it" state we're in now?

[psoares-resilient]

I can confirm the same started happening to us last Thursday. All of a sudden the same box that I was connected and tunneling decided to go ballistic and since all the other bastions started doing the same. As @bedge says, even the ones with agents that we have compiled ourselves.

An error occurred (BadRequest) when calling the StartSession operation: Currently, port forwarding sessions to remote hosts are not supported.

[neontty]

First of all I want to thank magJ and all of the AWS team for working hard to get this feature implemented.

Like others, I have deployed a solution that relies on this feature. Now I'm facing backlash on attempting to utilize an AWS-associated service instead of a manually configured bastion host.

Hello everyone,

We are still working on an internal review to release an AWS official document for this feature. We will keep you informed when there is an update.

Thanks, Jeremy

@jeremychangy Would you be able to give us a ballpark timeline on the internal review? If I can't give an estimate on when this feature will be turned back on, then we will likely abandon SSM because we will have to implement our second-choice solution to get our systems back running. SSM is much cleaner and I would prefer to use this new feature.

[psoares-resilient]

Not the news I was hoping for, but AWS support came back with:

"(...) service team and they've advised that the change needed further review. While I cannot share an exact ETA on this, we are hoping to have this pushed before the end of May 2022. This date is subject to change. (...)"

It was effectively a kill switch turned on since the release of this agent. If anyone manages to get it working (even on self-compiled agents), please let me know! Meanwhile, we will go back to the SOCAT workaround...

[ronkorving]

@jeremychangy After a month and a half, just hoping you could give us an update. Even no news is news at this point. Thanks.

[justinmk3]

@ronkorving does this discussion help? https://github.com/aws/amazon-ssm-agent/issues/208#issuecomment-1086766156

[neontty]

@justinmk3 unfortunately, no

[jrmy1]

Hello everyone,

We have now launched the remote host port forwarding feature in all the classic regions for agent versions 3.1.1374.0 and beyond. We sincerely apologize for the delay in releasing this feature. We rigorously test any update to the SSM agent to meet our high security bar – this took longer than we originally expected. Thanks for your patience and support; please do let us know your feedback.

Thanks, Jeremy

[quiver]

It's announced now. 🎉

AWS Systems Manager announces support for port forwarding to remote hosts using Session Manager