24.0.7 introduced a regression in network stuff

[djeanprost]

Hello,

I've referenced version 24 of docker image and dind image when executing a pipeline in gitlab runner. Suddenly, I started to get what I would call network issues. My pipeline started to run 24.0.7 after 24.0.6 When I run in 24.0.7, I get network issues I cannot explain. Reverting to 24.0.6 does the job. I cannot explain what changed between those two versions, but something related to network changed.

What fails is that DNS doesn't work anymore

RUN nslookup www.google.com ||true
;; connection timed out; no servers could be reached

Test KO with 24.0.7 :

#4 [1/9] FROM registry-pull.foo.fr/busybox@sha256:5c63a9b46e7139d2d5841462859edcbbf57f238af891b6096578e5894cfe5ae2
#4 resolve registry-pull.foo.fr/busybox@sha256:5c63a9b46e7139d2d5841462859edcbbf57f238af891b6096578e5894cfe5ae2 0.1s done
#4 extracting sha256:4e6669f810e6eb324aa88bc3ab1565d30e65e35dedca5260329c1be36951fe03
#4 sha256:f5fb98afcf9f5c6e8e069557f605b15b52643166c82ac5695f49fc6b0be04ee8 581B / 581B done
#4 sha256:4e6669f810e6eb324aa88bc3ab1565d30e65e35dedca5260329c1be36951fe03 2.22MB / 2.22MB 0.1s done
#4 sha256:5c63a9b46e7139d2d5841462859edcbbf57f238af891b6096578e5894cfe5ae2 9.52kB / 9.52kB done
#4 sha256:058f0df5310fbbbfea7e81a3a3e2b4bf3452438ec841138d170e170adbbd27a4 858B / 858B done
#4 extracting sha256:4e6669f810e6eb324aa88bc3ab1565d30e65e35dedca5260329c1be36951fe03 0.1s done
#4 DONE 0.5s
#5 [2/9] RUN cat /etc/resolv.conf
#5 0.349 # Generated by NetworkManager
#5 0.349 search foo.local
#5 0.349 nameserver 192.168.100.254
#5 DONE 0.5s
#6 [3/9] RUN ip route show ||true
#6 0.334 default via 172.17.0.1 dev eth0 
#6 0.334 172.17.0.0/16 dev eth0 scope link  src 172.17.0.2 
#6 DONE 0.4s
#7 [4/9] RUN nslookup www.google.com ||true
#7 5.312 ;; connection timed out; no servers could be reached
#7 5.312 
#7 DONE 5.9s

Test OK with 24.0.6

#4 [1/5] FROM registry-pull.foo.fr/busybox@sha256:5c63a9b46e7139d2d5841462859edcbbf57f238af891b6096578e5894cfe5ae2
#4 resolve registry-pull.foo.fr/busybox@sha256:5c63a9b46e7139d2d5841462859edcbbf57f238af891b6096578e5894cfe5ae2 0.0s done
#4 extracting sha256:4e6669f810e6eb324aa88bc3ab1565d30e65e35dedca5260329c1be36951fe03
#4 extracting sha256:4e6669f810e6eb324aa88bc3ab1565d30e65e35dedca5260329c1be36951fe03 0.1s done
#4 sha256:5c63a9b46e7139d2d5841462859edcbbf57f238af891b6096578e5894cfe5ae2 9.52kB / 9.52kB done
#4 sha256:058f0df5310fbbbfea7e81a3a3e2b4bf3452438ec841138d170e170adbbd27a4 858B / 858B done
#4 sha256:f5fb98afcf9f5c6e8e069557f605b15b52643166c82ac5695f49fc6b0be04ee8 581B / 581B done
#4 sha256:4e6669f810e6eb324aa88bc3ab1565d30e65e35dedca5260329c1be36951fe03 2.22MB / 2.22MB 0.1s done
#4 DONE 0.3s
#5 [2/5] RUN cat /etc/resolv.conf
#5 0.237 # Generated by NetworkManager
#5 0.237 search foo.local
#5 0.237 nameserver 192.168.100.254
#5 DONE 0.3s
#6 [3/5] RUN ip route show ||true
#6 0.278 default via 172.17.0.1 dev eth0 
#6 0.278 172.17.0.0/16 dev eth0 scope link  src 172.17.0.2 
#6 DONE 0.3s
#7 [4/5] RUN nslookup www.google.com ||true
#7 0.268 Server:        192.168.100.254
#7 0.268 Address:   192.168.100.254:53
#7 0.268 
#7 0.268 Non-authoritative answer:
#7 0.268 Name:  www.google.com
#7 0.268 Address: 216.58.215.36
#7 0.268 
#7 0.268 Non-authoritative answer:
#7 0.268 Name:  www.google.com
#7 0.268 Address: 2a00:1450:4007:80d::2004
#7 0.268 
#7 DONE 0.3s

[tianon]

What's your runner's host OS? (see #466 and #467, one of which this is probably a duplicate of, and #468 which is where we're trying to workshop fixes)

[alexislefebvre]

For us, the last image of dind caused no issues on runners that are up-to-date. But we also had the network issue when using docker:dind as a service in GitLab CI, on an older server.

Output of docker --version on the host of the GitLab runner:

• Docker version 20.10.24+dfsg1, build 297e128: no issue
• Docker version 20.10.7, build f0df350: no network

Pinning the image to docker:20.10.7-dind fixed the network issue.

[tianon]

Yeah, that matches one of #466 and #467, so I'm going to close this in favor of those two :innocent: :heart: