search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 117 forks source link

Docker-Desktop 4.27.0 breaks Istio on Kubernetes #7148

Closed xvzf closed 9 months ago

xvzf commented 9 months ago

Description

After the upgrade to 4.27.0 Istio running on-top of Docker Desktop Kubernetes stops working on the istio-init fails to start.

Here are the logs of one of the failing init containers:

2024-01-29T09:09:56.603820Z info    Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=true
INVALID_DROP=

2024-01-29T09:09:56.603868Z info    Istio iptables variables:
IPTABLES_VERSION=
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=5671,5672,25672
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=true
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[10.96.0.10],[]
NETWORK_NAMESPACE=
CNI_MODE=false
EXCLUDE_INTERFACES=

2024-01-29T09:09:56.603950Z info    Running iptables-restore with the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 5671 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 5672 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 25672 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp -m multiport ! --dports 53,15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp ! --dport 53 -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp ! --dport 53 -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -p tcp --dport 53 -d 10.96.0.10/32 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A OUTPUT -p udp --dport 53 -m owner --uid-owner 1337 -j RETURN
-A OUTPUT -p udp --dport 53 -m owner --gid-owner 1337 -j RETURN
-A OUTPUT -p udp --dport 53 -d 10.96.0.10/32 -j REDIRECT --to-port 15053
COMMIT
* raw
-A OUTPUT -p udp --dport 53 -m owner --uid-owner 1337 -j CT --zone 1
-A OUTPUT -p udp --sport 15053 -m owner --uid-owner 1337 -j CT --zone 2
-A OUTPUT -p udp --dport 53 -m owner --gid-owner 1337 -j CT --zone 1
-A OUTPUT -p udp --sport 15053 -m owner --gid-owner 1337 -j CT --zone 2
-A OUTPUT -p udp --dport 53 -d 10.96.0.10/32 -j CT --zone 2
-A PREROUTING -p udp --sport 53 -s 10.96.0.10/32 -j CT --zone 1
COMMIT
2024-01-29T09:09:56.604012Z info    Running command (with wait lock): iptables-restore --noflush --wait=30
2024-01-29T09:09:56.604934Z error   Command error output: xtables other problem: line 2 failed
2024-01-29T09:09:56.605120Z info    Running command (without lock): iptables-save 
2024-01-29T09:09:56.606686Z info    Command output: 
# Generated by iptables-save v1.8.7 on Mon Jan 29 09:09:56 2024
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Mon Jan 29 09:09:56 2024
# Generated by iptables-save v1.8.7 on Mon Jan 29 09:09:56 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A OUTPUT -p udp -m udp --dport 53 -m owner --uid-owner 1337 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -m owner --gid-owner 1337 -j RETURN
-A OUTPUT -d 10.96.0.10/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 5671 -j RETURN
-A ISTIO_OUTPUT -p tcp -m tcp --dport 5672 -j RETURN
-A ISTIO_OUTPUT -p tcp -m tcp --dport 25672 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m multiport ! --dports 53,15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 10.96.0.10/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Mon Jan 29 09:09:56 2024

2024-01-29T09:09:56.606706Z error   exit status 1

Reproduce

  1. istioctl install
  2. Start any pod with the istio sidecar enabled

Expected behavior

Istio works fine like in all previous releases

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:06:12 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.27.0 (135262)
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:35 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/xvzf/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.3-desktop.1
    Path:     /Users/xvzf/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.22
    Path:     /Users/xvzf/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/xvzf/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/xvzf/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/xvzf/.docker/cli-plugins/docker-scout

Server:
 Containers: 127
  Running: 68
  Paused: 0
  Stopped: 59
 Images: 47
 Server Version: 25.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.756GiB
 Name: docker-desktop
 ID: b3867f20-46f6-4d96-bb14-2840a648273a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

89814F6E-F5C8-4F6E-BCA8-FE138807A3C4/20240129091521

Additional Info

No response

xvzf commented 9 months ago

Note: this is also affecting Docker Desktop for Linux.

xvzf commented 9 months ago

The culprit is the kernel configuration, which disables a bunch of conntrack features required to run istio and other service meshes: 

-CONFIG_NF_CONNTRACK_ZONES=y
-CONFIG_NF_CONNTRACK_PROCFS=y
-CONFIG_NF_CONNTRACK_EVENTS=y
-CONFIG_NF_CONNTRACK_TIMEOUT=y
-CONFIG_NF_CONNTRACK_TIMESTAMP=y
+# CONFIG_NF_CONNTRACK_ZONES is not set
+# CONFIG_NF_CONNTRACK_PROCFS is not set
+# CONFIG_NF_CONNTRACK_EVENTS is not set
+# CONFIG_NF_CONNTRACK_TIMEOUT is not set
+# CONFIG_NF_CONNTRACK_TIMESTAMP is not set

For the record, the full diff can be found here: https://gist.github.com/xvzf/7a81f881033cab9a930f081929c3c5f0

dgageot commented 9 months ago

Thanks a lot @xvzf for fine the root cause. I'll add them back and add a few tests specific to istio.

dgageot commented 9 months ago

@xvzf do you know if only those 5 are required? How can I quickly test Istio nowadays? I haven't tried in a long time.

xvzf commented 9 months ago

Hi @dgageot, thanks for looking into this!

There's a platform prerequisites section in the Istio documentation but no specific configuration options for the modules (e.g. zone support on conntrack).

As for testing this specific bug, it should be sufficient to:

  1. Get a copy of istioctl ()
  2. Install istio e.g. with the demo profile using istioctl install --set profile=demo
  3. Label e.g. the default namespace with istio-injection=enabled
  4. Start any pod in the default namespace
  5. Validate the istio-init (init) container and istio-proxy container both come up

Also a suggestion form my colleague @jordiclariana:

docker run --rm -ti --cap-add NET_ADMIN gcr.io/istio-release/proxyv2:1.20.2 istio-iptables --redirect-dns

works on 4.61.1 but fails on 4.27.0. Might be worth adding it to your test-suite

dgageot commented 9 months ago

Hi @xvzf! Docker Desktop 4.27.1 is out. Could you tell me if it solves your issue?

dgageot commented 9 months ago

@xvzf Sorry for the false information. I just learnt that those patches will ship with 4.27.2

craigbox commented 9 months ago

David is back at Docker - hooray!

dgageot commented 9 months ago

David is back at Docker - hooray!

Indeed :-) Have you tried to start Istio with Docker Desktop 4.27.2, Craig?

craigbox commented 9 months ago

No, I just chanced into this link in the Istio slack.

I happen to have upgraded to 4.27.1 today but I had to do it manually - as the release notes suggested.

jordiclariana commented 9 months ago

@dgageot , we did try 4.27.2 with Istio and so far so good. Seems that the problem is solved there. Thanks!

dgageot commented 9 months ago

\o/ Thanks @jordiclariana. I'm closing the issue then