conmon keeps container ports opened after container death

[mildred]

I'm trying to use podman-compose to develop a compose file for an application, and I'm having issues when I try to start single containers after podman-compose up failed. I'm having an issue like that:

$ sudo podman-compose start web
using podman version: podman version 1.8.2
podman start myapp_web_1
Error: unable to start container "myapp_web_1": cannot listen on the TCP port: listen tcp4 :3000: bind: address already in use
125

And when I look, port 3000 is held by conmon that keeps running for the previous version of the same container that exited long ago.

I believe conmon should not keep opened ports for stopped containers.

Version information (all running on Fedora 32 silverblue):

• podman-compose-0.1.7.dev0
• podman-1.8.2-2.fc32.x86_64
• conmon-2.0.14-1.fc32.x86_64

[haircommander]

interesting that the port is kept open in conmon, as conmon didn't open them. The port ownership must have been reparented to conmon when the container died.

Thanks for the issue, I'll try to look at it this week.

[mheon]

We're actually passing file descriptors for the ports on the host into Conmon by default on the Podman side (as root, at least). CNI does its port-forwarding magic entirely in iptables, and as a result it's perfectly possible to have port 80 forwarded to a container and still have someone open it on the host (where it will receive no traffic, because iptables is routing it into the container). To try and prevent this, we open all ports forwarded to the container on the host in Podman, and then pass them into Conmon as extra FDs that will be closed when Conmon itself shuts down.

[haircommander]

@mildred in order to reproduce, I think I need to have your podman compose file. The real issue here is that conmon is not exiting when the container is exiting (or is exiting before it cleans up the fds)

[mildred]

That's a bit tricky as it involves source code I cannot give access to. Basically it involves running podman-compose as root on a Rails app (pretty standard stack). Some containers are failing to start and in the end I get conmon holding port 80 while there is no container running on that port.

[haircommander]

If you are able to get me a simple reproducer that doesn't show any secrets it would aid a lot in my debugging. I'll try my best otherwise

[dusansimic]

I've encountered the same problem when running tests for Gnomock. Since Gnomock connects to docker.sock to start and stop containers, I've needed to create podman.sock with sudo system service -t 0. Also I've installed podman-docker package which links /run/docker.sock to /run/podman/podman.sock. This should all work fine but when I run tests I get errors that a port is already in use. I've established that the same thing happens here.

It's not what you would call a simple reproducer but I hope it helped.

[polski-g]

I was able to reproduce this (once) via the following. Very strange how it is seemingly random.

podman run --rm -it -p 8080:80 -d --name asdf rockylinux:9 /bin/bash
podman stop asdf

lsof -Pni | grep 8080

conmon    28785 root    5u  IPv4 232895      0t0  TCP *:8080 (LISTEN)
3         28789 root    5u  IPv4 232895      0t0  TCP *:8080 (LISTEN)