mottati
commented
3 years ago I did a little more research on this. It appears that our use of the "local" network name is somehow triggering a different execution path within docker. In order to test this, I ran 3 more tests, similar to the ones in the bug report.
Prior to running these tests I created a second overlay network, created identically to the way the "local" network was created. Here is how it was created, and what it looked like.
$ docker network create -d overlay --attachable overlay-2
ujimx2wmy4g5806y9yxpc9eqq
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
9c44f239b788 bridge bridge local
dde755403625 docker_gwbridge bridge local
857e6f415c0d host host local
kybp70o5higc ingress overlay swarm
z5r05znqats7 local overlay swarm
cc1a68b2c2a3 none null local
ujimx2wmy4g5 overlay-2 overlay swarmTest 1: run the container using standard network assignment and connect back to Docker Host
$ docker run -it ubuntu-container ssh s-mxq80106bp.sys.az1.eng.pdx.wd
The authenticity of host 's-mxq80106bp.sys.az1.eng.pdx.wd (10.96.32.175)' can't be established.
ECDSA key fingerprint is SHA256:C47QAI27GeAx+M3/GxAGMoSi5RlFyiRtpFvdRoERXWk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? noTest 2: Repeat test 1 using newly created overlay-2 network
$ docker run -it --network overlay-2 ubuntu-container ssh s-mxq80106bp.sys.az1.eng.pdx.wd
The authenticity of host 's-mxq80106bp.sys.az1.eng.pdx.wd (10.96.32.175)' can't be established.
ECDSA key fingerprint is SHA256:C47QAI27GeAx+M3/GxAGMoSi5RlFyiRtpFvdRoERXWk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? noTest 3: Repeat test 1 using local network
$ docker run -it --network local ubuntu-container ssh s-mxq80106bp.sys.az1.eng.pdx.wd
ssh: connect to host s-mxq80106bp.sys.az1.eng.pdx.wd port 22: Connection refusedAs can be seen above, only the third test fails leading me to suspect that there is some kind of special behavior associated with the overlay network called "local".
The ubuntu-container used in this test was created from a jetbrains/teamcity-agent:2021.1.2 base.
TCP connection attempts from a Swarm hosted container are unable to make connections to the hostname FQDN of the docker host. Connections from the container to the same host via the IP address or a CNAME do connect.
Expected behavior
Connection via the FQDN hostname of the Docker Host should connect
Actual behavior
Connection to hostname FQDN is refused.
Steps to reproduce the behavior
Environment Info
10.96.32.175
From the docker host, exec into a container and from that container make an ssh connection back to the docker host. Attempt this connection in three different ways. First using the IP address of the Docker host, next using the CNAME that refers to the Docker host, last using the FQDN of the docker host.
The first two connection attempts work, the third fails.
Connect with IP address
Connect with CNAME
Connect with Docker Host FQDN
The only connection is refused is the attempt to connect to the FQDN of the docker host. Every other mechanism that would resolve down to the same IP address works. I have included the network layout below. One thing to note is that we are using the "local" network as an overlay connecting our dispirat stacks. We do this so that connection attempts to.local do not leak out to the internet when the container does not exist. See: https://en.wikipedia.org/wiki/.local
We began to see this issue when we upgraded or Docker version from 19.3.11 to 20.10.8
Output of docker network ls
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.)