`cilium connectivity test` can't resolve host: one.one.one.one

MerzMax commented 1 year ago

Bug report

General Information

Cilium CLI version (run cilium version)

$ cilium version
cilium-cli: v0.12.8 compiled with go1.19.3 on linux/amd64
cilium image (default): v1.12.2
cilium image (stable): v1.12.4
cilium image (running): v1.12.4

Orchestration system version in use (e.g. kubectl version, ...)

$ kubectl version --short
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.25.4

Platform / infrastructure information (e.g. AWS / Azure / GCP, image / kernel versions)

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:    22.04
Codename:   jammy

Link to relevant artifacts (policies, deployments scripts, ...)

used helm to deploy cilium

$ helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}

Generate and upload a system zip: cilium-sysdump-20221129-145024.zip

How to reproduce the issue

Execute cilium connectivity test

$ cilium connectivity test
ℹ️  Single-node environment detected, enabling single-node connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
⌛ [kubernetes] Waiting for deployments [client client2 echo-same-node] to become ready...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/client-7db976bfbf-k245w to appear...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/client2-6f8b754559-k58xx to appear...
⌛ [kubernetes] Waiting for pod cilium-test/client2-6f8b754559-k58xx to reach DNS server on cilium-test/echo-same-node-6d59fd9bc4-6f8wq pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-7db976bfbf-k245w to reach DNS server on cilium-test/echo-same-node-6d59fd9bc4-6f8wq pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-7db976bfbf-k245w to reach default/kubernetes service...
⌛ [kubernetes] Waiting for pod cilium-test/client2-6f8b754559-k58xx to reach default/kubernetes service...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-6d59fd9bc4-6f8wq to appear...
⌛ [kubernetes] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [kubernetes] Waiting for NodePort 10.100.255.87:30636 (cilium-test/echo-same-node) to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.12.4
🏃 Running tests...

[=] Test [no-policies]
....................
[=] Test [allow-all-except-world]
........
[=] Test [client-ingress]
..
[=] Test [all-ingress-deny]
......
[=] Test [all-egress-deny]
........
[=] Test [all-entities-deny]
......
[=] Test [cluster-entity]
..
[=] Test [host-entity]
..
[=] Test [echo-ingress]
..
[=] Test [client-ingress-icmp]
..
[=] Test [client-egress]
..
[=] Test [client-egress-expression]
..
[=] Test [client-egress-to-echo-service-account]
..
[=] Test [to-entities-world]
.
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-to-entities-world' to namespace 'cilium-test'..
  [-] Scenario [to-entities-world/pod-to-world]
  [.] Action [to-entities-world/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code 6
  ℹ️  curl output:
  curl: (6) Could not resolve host: one.one.one.one
:0 -> :0 = 000

  📄 No flows recorded during action http-to-one-one-one-one-0
  📄 No flows recorded during action http-to-one-one-one-one-0
  [.] Action [to-entities-world/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-0
  📄 No flows recorded during action https-to-one-one-one-one-0
  [.] Action [to-entities-world/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443/index.html" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-index-0
  📄 No flows recorded during action https-to-one-one-one-one-index-0
  [.] Action [to-entities-world/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code 6
  ℹ️  curl output:
  curl: (6) Could not resolve host: one.one.one.one
:0 -> :0 = 000

  📄 No flows recorded during action http-to-one-one-one-one-1
  📄 No flows recorded during action http-to-one-one-one-one-1
  [.] Action [to-entities-world/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-1
  📄 No flows recorded during action https-to-one-one-one-one-1
  [.] Action [to-entities-world/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443/index.html" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-index-1
  📄 No flows recorded during action https-to-one-one-one-one-index-1
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-to-entities-world' from namespace 'cilium-test'..

[=] Test [to-cidr-1111]
....
[=] Test [echo-ingress-l7]
......
[=] Test [echo-ingress-l7-named-port]
......
[=] Test [echo-ingress-from-other-client-deny]
....
[=] Test [client-ingress-from-other-client-icmp-deny]
....
[=] Test [client-egress-to-echo-deny]
....
[=] Test [client-ingress-to-echo-named-port-deny]
..
[=] Test [client-egress-to-echo-expression-deny]
..
[=] Test [client-egress-to-echo-service-account-deny]
..
[=] Test [client-egress-to-cidr-deny]
....
[=] Test [client-egress-to-cidr-deny-default]
....
[=] Test [health]
.
[=] Test [client-egress-l7-method]
......
[=] Test [client-egress-l7]
...
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7/pod-to-pod]
  [.] Action [client-egress-l7/pod-to-pod/curl-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> cilium-test/echo-same-node-6d59fd9bc4-6f8wq (10.0.0.80:8080)]
  [.] Action [client-egress-l7/pod-to-pod/curl-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> cilium-test/echo-same-node-6d59fd9bc4-6f8wq (10.0.0.80:8080)]
  [-] Scenario [client-egress-l7/pod-to-world]
  [.] Action [client-egress-l7/pod-to-world/http-to-one-one-one-one-0: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code 6
  ℹ️  curl output:
  curl: (6) Could not resolve host: one.one.one.one
:0 -> :0 = 000

  📄 No flows recorded during action http-to-one-one-one-one-0
  📄 No flows recorded during action http-to-one-one-one-one-0
  [.] Action [client-egress-l7/pod-to-world/https-to-one-one-one-one-0: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)]
  [.] Action [client-egress-l7/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)]
  [.] Action [client-egress-l7/pod-to-world/http-to-one-one-one-one-1: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)]
  [.] Action [client-egress-l7/pod-to-world/https-to-one-one-one-one-1: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)]
  [.] Action [client-egress-l7/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)]
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-only-dns' from namespace 'cilium-test'..
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-l7-http' from namespace 'cilium-test'..

[=] Test [client-egress-l7-named-port]
......
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7-named-port/pod-to-pod]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> cilium-test/echo-same-node-6d59fd9bc4-6f8wq (10.0.0.80:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> cilium-test/echo-same-node-6d59fd9bc4-6f8wq (10.0.0.80:8080)]
  [-] Scenario [client-egress-l7-named-port/pod-to-world]
  [.] Action [client-egress-l7-named-port/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code 6
  ℹ️  curl output:
  curl: (6) Could not resolve host: one.one.one.one
:0 -> :0 = 000

  📄 No flows recorded during action http-to-one-one-one-one-1
  📄 No flows recorded during action http-to-one-one-one-one-1
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)]
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-only-dns' from namespace 'cilium-test'..
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-l7-http-named-port' from namespace 'cilium-test'..

[=] Test [dns-only]
...
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  [-] Scenario [dns-only/pod-to-pod]
  [.] Action [dns-only/pod-to-pod/curl-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> cilium-test/echo-same-node-6d59fd9bc4-6f8wq (10.0.0.80:8080)]
  [.] Action [dns-only/pod-to-pod/curl-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> cilium-test/echo-same-node-6d59fd9bc4-6f8wq (10.0.0.80:8080)]
  [-] Scenario [dns-only/pod-to-world]
  [.] Action [dns-only/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action http-to-one-one-one-one-0
  📄 No flows recorded during action http-to-one-one-one-one-0
  [.] Action [dns-only/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-0
  📄 No flows recorded during action https-to-one-one-one-one-0
  [.] Action [dns-only/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443/index.html" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-index-0
  📄 No flows recorded during action https-to-one-one-one-one-index-0
  [.] Action [dns-only/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action http-to-one-one-one-one-1
  📄 No flows recorded during action http-to-one-one-one-one-1
  [.] Action [dns-only/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-1
  📄 No flows recorded during action https-to-one-one-one-one-1
  [.] Action [dns-only/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443/index.html" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-index-1
  📄 No flows recorded during action https-to-one-one-one-one-index-1
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-only-dns' from namespace 'cilium-test'..

[=] Test [to-fqdns]
.
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-to-fqdns-one-one-one-one' to namespace 'cilium-test'..
  [-] Scenario [to-fqdns/pod-to-world]
  [.] Action [to-fqdns/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code 6
  ℹ️  curl output:
  curl: (6) Could not resolve host: one.one.one.one
:0 -> :0 = 000

  📄 No flows recorded during action http-to-one-one-one-one-0
  📄 No flows recorded during action http-to-one-one-one-one-0
  [.] Action [to-fqdns/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-0
  📄 No flows recorded during action https-to-one-one-one-one-0
  [.] Action [to-fqdns/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443/index.html" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-index-0
  📄 No flows recorded during action https-to-one-one-one-one-index-0
  [.] Action [to-fqdns/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null http://one.one.one.one:80" failed: command terminated with exit code 6
  ℹ️  curl output:
  curl: (6) Could not resolve host: one.one.one.one
:0 -> :0 = 000

  📄 No flows recorded during action http-to-one-one-one-one-1
  📄 No flows recorded during action http-to-one-one-one-one-1
  [.] Action [to-fqdns/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-1
  📄 No flows recorded during action https-to-one-one-one-one-1
  [.] Action [to-fqdns/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --connect-timeout 5 --output /dev/null https://one.one.one.one:443/index.html" failed with unexpected exit code: command terminated with exit code 6 (expected 28, found 6)
  📄 No flows recorded during action https-to-one-one-one-one-index-1
  📄 No flows recorded during action https-to-one-one-one-one-index-1
  [-] Scenario [to-fqdns/pod-to-world-2]
  [.] Action [to-fqdns/pod-to-world-2/https-cilium-io-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> cilium-io-https (cilium.io:443)]
  [.] Action [to-fqdns/pod-to-world-2/https-cilium-io-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> cilium-io-https (cilium.io:443)]
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-egress-to-fqdns-one-one-one-one' from namespace 'cilium-test'..

📋 Test Report
❌ 5/31 tests failed (20/151 actions), 0 tests skipped, 1 scenarios skipped:
Test [to-entities-world]:
  ❌ to-entities-world/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)
  ❌ to-entities-world/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)
  ❌ to-entities-world/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)
  ❌ to-entities-world/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)
  ❌ to-entities-world/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)
  ❌ to-entities-world/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)
Test [client-egress-l7]:
  ❌ client-egress-l7/pod-to-world/http-to-one-one-one-one-0: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)
Test [client-egress-l7-named-port]:
  ❌ client-egress-l7-named-port/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)
Test [dns-only]:
  ❌ dns-only/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)
  ❌ dns-only/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)
  ❌ dns-only/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)
  ❌ dns-only/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)
  ❌ dns-only/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)
  ❌ dns-only/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)
Test [to-fqdns]:
  ❌ to-fqdns/pod-to-world/http-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-http (one.one.one.one:80)
  ❌ to-fqdns/pod-to-world/https-to-one-one-one-one-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https (one.one.one.one:443)
  ❌ to-fqdns/pod-to-world/https-to-one-one-one-one-index-0: cilium-test/client-7db976bfbf-k245w (10.0.0.150) -> one-one-one-one-https-index (one.one.one.one:443)
  ❌ to-fqdns/pod-to-world/http-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-http (one.one.one.one:80)
  ❌ to-fqdns/pod-to-world/https-to-one-one-one-one-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https (one.one.one.one:443)
  ❌ to-fqdns/pod-to-world/https-to-one-one-one-one-index-1: cilium-test/client2-6f8b754559-k58xx (10.0.0.193) -> one-one-one-one-https-index (one.one.one.one:443)
connectivity test failed: 5 tests failed

As you can see from the output curl can't reesolve the host one.one.one.one. That's why 5/31 tests fail.

After some research we now have an idea of what's going on. For the tests the base image is an Alpine image (see here). For some reason Alpine has problems with DNS resolution in kubernetes clusters in it's musl library. Here you can find a very good explaination of what is happening:

Stack Overeflow entry

brb commented 1 year ago

@MerzMax Thanks for the issue. How did you come to the conclusion that the DNS resolution but no the actual connection to the 1.1.1.1 did fail? Did you check the Hubble flow logs? We have observed that sometimes connections to the 1.1.1.1 fail, see for more details here - https://cilium.slack.com/archives/C7PE7V806/p1668619257856639. I think this issue is yet another instance of the same connectivity failure.

MerzMax commented 1 year ago

@brb The issue linked in the Slack message describes a timeout when curl is executed. In my case the hostname one.one.one.one can't get resolved.

It has to be a DNS issue since I am able to connect to 1.1.1.1 but not to one.one.one.one. What is possible is to connect to one.one.one.one., what shows the issue described by the Stack Overflow entry linked above.

Here is the output I get when connecting in the client pod and executing curl:

$ kubectl exec -it client2-6f8b754559-k58xx sh -n cilium-test
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ # curl 1.1.1.1
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
/ # curl one.one.one.one
curl: (6) Could not resolve host: one.one.one.one
/ # curl one.one.one.one.
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
/ #

sqlstatement commented 1 year ago

Just encountered the same issue. In my case, it was solved by removing the search domain from the host.

cdfpaz commented 1 year ago

kudos @sqlstatement, lost 2 days of work rewriting my cluster/cilium conf, until reach your answer :)

erikschul commented 1 year ago

@sqlstatement Could you elaborate? I'm new to Kubernetes. I've already spent hours trying to debug this issue. @bzero Or do you have any suggestions?

sqlstatement commented 1 year ago

@erikschul Your search domain is probably handled by either:

the DHCP client (if you use DHCP)
/etc/network/interfaces
/etc/resolv.conf

Once you remove the search domain, the connectivity test should run as expected. Hope this helps :)

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

github-actions[bot] commented 2 weeks ago

This issue has not seen any activity since it was marked stale. Closing.

cilium / cilium-cli

`cilium connectivity test` can't resolve host: one.one.one.one #1243

Bug report