search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 118 forks source link

localhost is resolved to `::1` (IPv6) instead of `127.0.0.1` #7269

Open GuLinux opened 4 months ago

GuLinux commented 4 months ago

Description

Since upgrading to docker desktop v4.29.0, we've noticed that the host resolution for localhost is resolving to the IPv6 version (::1) instead of IPv4 (127.0.0.1).

Specifically, with a nslookup I can notice that the order of IPv4 and IPv6 resolution is inverted, and in v4.29.0 the latter is prioritised.

This is not mentioned in the changelog, so I assume it's a bug, especially since IPv6 networks are not supported on Docker for Mac. In our environment services start with IPv4 stack, so having healthcheck calls to http://localhost/status result to the containers status being "unhealthy".

Reproduce

Non-authoritative answer: Name: localhost Address: 127.0.0.1

Non-authoritative answer: Name: localhost Address: ::1

/ # ping localhost PING localhost (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.330 ms

 - Install Docker Desktop v4.29.0
 - Run:

docker run -it --rm alpine sh / # nslookup localhost Server: 192.168.65.7 Address: 192.168.65.7:53

Non-authoritative answer: Name: localhost Address: ::1

Non-authoritative answer: Name: localhost Address: 127.0.0.1

/ # ping localhost PING localhost (::1): 56 data bytes 64 bytes from ::1: seq=0 ttl=64 time=0.330 ms


### Expected behavior

localhost should resolve to the IPv4 address (`127.0.0.1`)

### docker version

```bash
Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:14:46 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:02 2024
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.0.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     /Users/mgu06/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     /Users/mgu06/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     /Users/mgu06/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/mgu06/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/mgu06/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/mgu06/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/mgu06/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/mgu06/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     /Users/mgu06/.docker/cli-plugins/docker-scout

Server:
 Containers: 8
  Running: 1
  Paused: 0
  Stopped: 7
 Images: 134
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 4.071GiB
 Name: docker-desktop
 ID: a5de42d5-7433-4f06-9b3c-110fa899822b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/mgu06/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

1947FF30-F1E0-456C-A352-91A4B412F253/20240501082456

Additional Info

No response

dgageot commented 4 months ago

cc @djs55

Sergio-bzh commented 4 months ago

Hi every one, I hope you're all doing well as your beloved ones.

I'm facing the same issue as GuLinux since last update :-(

robmry commented 4 months ago

Thank you for the nice clear report and examples - I think I see what's happening.

To give some background ...

In earlier releases, the Docker engine would only enable IPv6 on the loopback interface when a container was connected to an IPv6-enabled network. When it was disabled, there was no ::1 address in the container. So, when a container was connected to its first IPv6-enabled network, or disconnected from the last, the ::1 address would appear or disappear.

But, the /etc/hosts file always contained an entry for ::1 localhost (and DNS would respond with ::1). That caused problems for some applications when there was no ::1 address.

That DNS response for an absent ::1 is what you're seeing with Docker Desktop v4.28.0. The nslookup command is making two requests, one for an A record and another for an AAAA record. Those requests race, and which response gets back to nslookup first is just luck. (You'll probably see the order change if you run the nslookup command a few times.)

As part of ongoing work to improve Docker's support for IPv6 - by default, we now leave IPv6 enabled on the loopback interface, so it always has a ::1 address. That makes network connect/disconnect more like connecting an ethernet cable to a physical host, the interface may or may not get an IPv6 address - but it doesn't cause reconfiguration of the host's loopback interface.

So ...

I think the change you're seeing is probably because ::1 is now present on the loopback interface.

If the ::1 address didn't exist, the healthcheck request went to 127.0.0.1 instead. (You're seeing the inverse of the problem caused by the absence of ::1.)

In which case, there are a couple of options ...

If the healthcheck only works on IPv4, it would probably be best to change the healthcheck URL to http://127.0.0.1/status.

Alternatively, you can disable IPv6 in the container so that it never has a ::1 (or an /etc/hosts entry for ::1). To do that, use --sysctl=net.ipv6.conf.all.disable_ipv6=1 in the docker run command. Or, the equivalent sysctls option in a compose file.

Hopefully that makes sense, and helps? Let me know!

GuLinux commented 4 months ago

Hi, thank you for your reply.

Yes, I thought of using 127.0.0.1 instead as a workaround, and yes, we could potentially alter /etc/hosts in the containers. Both of these look like workarounds though, I was hoping to have an actual solution in Docker itself, as this is clearly a breaking change (I wouldn't expect every single image in docker hub to have their respective /etc/hosts file altered because of this)

robmry commented 4 months ago

The /etc/hosts file is generated by the engine and mounted into the container as it starts up. It has always included an entry for ::1.

Editing the /etc/hosts file in a running container, removing IPv6 entries if not required, may be an option. But yes, it would be a fiddly workaround (so I didn't mention it before).

The change is that the loopback interface now keeps the ::1 address, even while the container is not connected to an IPv6 network. Removing ::1 from the interface, depending on which networks were connected, was problematic and unusual behaviour - very unlike a Linux host. The change to the new default behaviour is needed as we make improvements to the way IPv6 is configured.

Explicitly using the IPv4 address for an IPv4-only service will always work.

The old behaviour, no ::1, can be restored for an IPv4-only container using --sysctl=net.ipv6.conf.all.disable_ipv6=1.

GuLinux commented 4 months ago

Thanks, if that's the case, we'll start using 127.0.0.1 as for now it's the easiest solution for us

Sergio-bzh commented 3 months ago

Hi you all, Thanks for explanations and workaround.

👍

tscully49 commented 3 months ago

Facing the same issue here. This broke some automation of ours and we lost hours of time to debugging.

sammyhk commented 3 months ago

Related to change of https://github.com/moby/moby/pull/47062 in Docker 26+

sammyhk commented 3 months ago

Thank you for the nice clear report and examples - I think I see what's happening.

To give some background ...

In earlier releases, the Docker engine would only enable IPv6 on the loopback interface when a container was connected to an IPv6-enabled network. When it was disabled, there was no ::1 address in the container. So, when a container was connected to its first IPv6-enabled network, or disconnected from the last, the ::1 address would appear or disappear.

But, the /etc/hosts file always contained an entry for ::1 localhost (and DNS would respond with ::1). That caused problems for some applications when there was no ::1 address.

That DNS response for an absent ::1 is what you're seeing with Docker Desktop v4.28.0. The nslookup command is making two requests, one for an A record and another for an AAAA record. Those requests race, and which response gets back to nslookup first is just luck. (You'll probably see the order change if you run the nslookup command a few times.)

As part of ongoing work to improve Docker's support for IPv6 - by default, we now leave IPv6 enabled on the loopback interface, so it always has a ::1 address. That makes network connect/disconnect more like connecting an ethernet cable to a physical host, the interface may or may not get an IPv6 address - but it doesn't cause reconfiguration of the host's loopback interface.

So ...

I think the change you're seeing is probably because ::1 is now present on the loopback interface.

If the ::1 address didn't exist, the healthcheck request went to 127.0.0.1 instead. (You're seeing the inverse of the problem caused by the absence of ::1.)

In which case, there are a couple of options ...

If the healthcheck only works on IPv4, it would probably be best to change the healthcheck URL to http://127.0.0.1/status.

Alternatively, you can disable IPv6 in the container so that it never has a ::1 (or an /etc/hosts entry for ::1). To do that, use --sysctl=net.ipv6.conf.all.disable_ipv6=1 in the docker run command. Or, the equivalent sysctls option in a compose file.

Hopefully that makes sense, and helps? Let me know!

I think the root cause is docker generated /etc/hosts with duplicated entry of localhost which can resolve to IPv4 127.0.0.1 or IPv6 ::1:

$ cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.0.5.6        1f0663d9fd4f

While a bare metal Linux machine will not have such duplicated localhost entry, reference from Ubuntu 22.04 bare metal IPv6 enabled server:

$ cat /etc/hosts
127.0.0.1   localhost
127.0.1.1   server-hostname

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
robmry commented 3 months ago

Hi @sammyhk - that's interesting, but not universal ...

Debian 12.5:

127.0.0.1   localhost
127.0.1.1   debian.myguest.virtualbox.org   debian

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

CentOS Stream 9:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

openSUSE Tumbleweed:

127.0.0.1   localhost localhost.localdomain
::1     localhost localhost.localdomain ipv6-localhost ipv6-loopback

# special IPv6 addresses
fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

Only having 127.0.0.1 localhost like Ubuntu should usually work. But, particularly as Docker has always included ::1, removing it now might be another breaking change for some users.

sammyhk commented 3 months ago

@robmry Hmm, interesting. I checked in Docker 24.0.7, as before https://github.com/moby/moby/pull/47062 , the /etc/hosts still contains the ::1 localhost entry but ping localhost always return 127.0.0.1. After that change in Docker 26+, ping localhost now always return ::1.

I really don't understand why have this behavior now...

robmry commented 3 months ago

I really don't understand why have this behavior now...

Is it not this? ... https://github.com/docker/for-mac/issues/7269#issuecomment-2107333702

sammyhk commented 3 months ago

Is it not this? ... #7269 (comment)

Yes, I checked after adding --sysctl=net.ipv6.conf.all.disable_ipv6=1 can revert back to the previous behavior, with one thing changed which /etc/hosts does not contains IPv6 stuff (Docker 26- will have those IPv6 stuff even no IPv6 enabled), which is good. What I don't understand is, even when I added back those IPv6 stuff in /etc/hosts, ping localhost can always resolved to 127.0.0.1 instead of ::1. To illustrate (--sysctl=net.ipv6.conf.all.disable_ipv6=1 set):

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
 ...

# ping localhost
PING localhost (127.0.0.1): 56 data bytes
...

# ping -6 localhost
ping: bad address 'localhost'

## after this line, edited /etc/hosts to add
::1 localhost ip6-localhost ip6-loopback

# ping localhost
PING localhost (127.0.0.1): 56 data bytes
...

# ping -6 localhost
PING localhost (::1): 56 data bytes
ping: sendto: Address not available

When in system which --sysctl=net.ipv6.conf.all.disable_ipv6=1 not set, ping localhost always resolve to ::1, seems like IPv6 have higher preceding than IPv4.

robmry commented 3 months ago

seems like IPv6 have higher preceding than IPv4.

Yes, that's RFC-6724. With glibc, the precedence is configurable.

Yaytay commented 2 months ago

We use swarm. Looking into what it will take to set the sysctl for swarm in our compose files I find that the spec (https://github.com/docker/compose/blob/v1/docs/Compose%20file%20reference%20(legacy)/version-3.md) says this: IPv6 options do not currently work in swarm mode.

I'm finding it hard to believe that https://github.com/moby/moby/pull/47062 solves more problems than it causes :(

Don't we need a way to disable IPv6 for all containers on a host?

robmry commented 2 months ago

IPv6 options do not currently work in swarm mode.

The --sysctl option isn't IPv6-specific ... docker service create --sysctl net.ipv6.conf.all.disable_ipv6=1 ... works, as well as the equivalent in a compose file:

services:
  myservice:
      sysctls:
        - net.ipv6.conf.all.disable_ipv6=1