Container ports are reassigned upon restart

[dmartin]

Challenge containers are launched with an always restart policy, but unfortunately Docker reassigns host ports upon restart when an explicit port mapping is not specified up front. This causes the port information in the database to become inaccurate if a container restarts.

For example:

$ cd ~/cmgr/examples

$ ../bin/cmgr update
# [...]

$ ../bin/cmgr build cmgr/examples/flask--sqlite 1
# Build IDs:
#    1

$ ../bin/cmgr start 1
# Instance ID: 1

$ docker container ls --format "table {{.ID}}\t{{.Image}}\t{{.Ports}}"
# CONTAINER ID   IMAGE                                     PORTS
# fad446c0acdf   cmgr/examples/flask--sqlite:1-challenge   0.0.0.0:61285->5000/tcp

$ docker restart fad446c  # same behavior if automatically restarted
# fad446c

$ docker container ls --format "table {{.ID}}\t{{.Image}}\t{{.Ports}}"
# CONTAINER ID   IMAGE                                     PORTS
# fad446c0acdf   cmgr/examples/flask--sqlite:1-challenge   0.0.0.0:61323->5000/tcp

[jrolli]

@dmartin, I think I'm leaning toward the idea of fixing this by allowing an explicit port range to be configured for challenge ports (likely via an environment variable). If the configuration is set, then cmgr/cmgrd will assume they fully control assignments in that port range and manage it through querying the database. If it is not set, then it continues to use the current behavior (delegating to Docker and using ephemeral ports). It would end up being a larger change, but I think is needed to avoid racing with other connections. Thoughts?

[dmartin]

@jrolli, I think that works. Perhaps it should also be mentioned in the documentation to adjust /proc/sys/net/ipv4/ip_local_port_range to exclude the block carved out for challenges, so that other applications' attempts to bind an ephemeral port don't land within the cmgr range (some services like dockerd etc. would need to be restarted as well, as it looks like they only read the ephemeral port range once upon init.)

An aside: looking into this sent me down a rabbit hole of looking into how Docker actually allocates ephemeral ports, which turns out to be... pretty poorly? If I'm understanding correctly, they don't use bind(2) to get an available port at all, but just maintain their own map and hope for the best. Actually asking the OS for an available port has been a TODO for 7 years(!)

[jrolli]

Fixed in v0.11.1