Accessing forwarded port from Docker container

[maxispeicher]

Hello,

I am using the PortForwardingSession to access some services in our AWS account. When using it locally everything works fine and I have access via localhost:<port>. However, inside a docker container I'm not able to access the same port. At the same time I am able to access a different port with a local web server without any issues. So the problem only seems to exist in combination of SSM and Docker. When using nmap from inside the container it also shows the port used for the forwarding session as closed.

docker --version -> 20.10.8
session-manager-plugin --version -> 1.2.245.0

running on Ubuntu 20.04.3 LTS

Do you know any reason why the SSM port behaves differently in comparison to e.g. a webserver.

Many Thanks!

[YujiaozhAws]

Thanks for reaching out! Could you please provide below information to investigate this issue further?

1. Is only one local port not accessible or are you not able to successfully use Session Manager inside the container?
2. Provide a few lines of debug log of Session Manager plugin.
3. Is there any port mapping for the container?

Reference: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-configure-logs

[sanjeevik89]

I am seeing the same problem that @maxispeicher raised in this issue.

1. Port Forwarding Session Manager is working inside container.
2. Attached the logs for reference.
3. port mapping is set for 8080. sessionmanager.txt

[maxispeicher]

Sorry for the late reply, but I kind of managed to "fix" it. However, the solution is not optimal. After changing the line 127.0.0.1 localhost to 0.0.0.0 localhost in /etc/hosts the forwarding also works inside the docker container.

[krystian-panek-vmltech]

also struggling with that problem but I am afraid to correct that /etc/hosts because I have some other software on that particular container on which I am using SSM

how about just making configurable also that localhost here? localHost param? https://github.com/aws/session-manager-plugin/blob/c523002ee02c8b68983ad05042ed52c44d867952/src/sessionmanagerplugin/session/portsession/basicportforwarding.go#L153

to be able to put there 0.0.0.0 ?:)

[dougmoscrop]

Hopefully the PR that fixes this will be merged soon, but in the meantime, if you're (rightfully) avoiding modifying /etc/hosts you could also use socat. If you have an ECS Task listening on port 8000, and want to connect to that from your host machine on port 8000, via a Docker-run SSM port forwarding session, it looks something like this:

docker (compose) run -p 8000:8000 [...] socat TCP-LISTEN:8000,fork,reuseaddr TCP:127.0.0.1:8081 & aws ssm start-session [...] --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["8000"],"localPortNumber":["8001"]}'

where [...] is your image that has the AWS CLI, SSM Plugin, and socat installed, AWS region/profile/target, etc.

[krystian-panek-vmltech]

yes I have used socat as workaround ;) @dougmoscrop

[anas-zurkiyeh-sp]

Any insight on when this will be fixed?

[josealdaco]

Any updates on the fix? :)

[fneco]

I don't know if this is related, but this works for me.

docker run -it --rm  \
  -e "PGPASSWORD=${DB_PASSWORD}" \
  postgres:16.3 psql -h host.docker.internal -p 5432 -U foo -d bar

The key is to specify host.docker.internal as the host.

[MikeJansen]

I have begun work on a PR for this. If anyone that has worked on this plugin can give me a jump start on understanding the networking aspect of the plugin, that would help.

Without delving too much into the big picture, I made a first pass attempt to get it working simply by overriding the hostname for the listener. See change.

The listener listens on the specified interface, port. I'll add more info later with details of what I'm encountering. I'm hoping to make initial contact with a maintainer at this point.

[audrey-roe]

If you've tried the solutions mentioned above without success, here's another approach that worked for me:

1. Check if you have PostgreSQL running locally on your machine.
2. If so, try stopping it. In my case, I had PostgreSQL 14 installed via Homebrew.
3. To stop the local PostgreSQL service, I ran: brew services stop postgresql
4. After stopping the local service, I retried connecting to the Docker PostgreSQL instance, and it worked.

Note: Changing 127.0.0.1 localhost to 0.0.0.0 localhost in /etc/hosts also worked for me, but it's not recommended as it can potentially cause other issues.

For more information on this approach, see this StackOverflow post.

Hopefully, this helps someone facing a similar issue!

[thiagomarafeli]

I'm running through the same problem here! It's painful to use a non-oficial solution just to get around that...

Any updates on this?? This would be a major and important feature