cilium connectivity test is deploying echo-external-node deployment while node-without-cilium is disabled

[sathieu]

Bug report

The cluster is correctly discovered with node-without-cilium disabled, but echo-external-node deployment is deployed anyway. Why?

 $ cilium connectivity test   --curl-insecure   --external-cidr 10.0.0.0/8   --external-target redacted.example.org   --external-ip 10.20.30.40   --external-other-ip 10.22.33.44 --debug
🐛 Detected features:
🐛   cilium-network-policy: Enabled
🐛   cni-chaining: Disabled:none
🐛   enable-ipv4-egress-gateway: Disabled
🐛   encryption-node: Disabled
🐛   encryption-pod: Disabled:disabled
🐛   endpoint-routes: Disabled
🐛   flavor: Enabled:unknown
🐛   health-checking: Enabled
🐛   host-firewall: Disabled
🐛   host-port: Enabled
🐛   icmp-policy: Enabled
🐛   ingress-controller: Disabled
🐛   ipv4: Enabled
🐛   ipv6: Disabled
🐛   k8s-network-policy: Enabled
🐛   kpr-external-ips: Enabled
🐛   kpr-graceful-termination: Enabled
🐛   kpr-hostport: Enabled
🐛   kpr-mode: Enabled:Strict
🐛   kpr-nodeport: Enabled
🐛   kpr-session-affinity: Enabled
🐛   kpr-socket-lb: Enabled
🐛   l7-proxy: Enabled
🐛   monitor-aggregation: Enabled:medium
🐛   mutual-auth-spiffe: Disabled
🐛   node-without-cilium: Disabled
🐛   secret-backend-k8s: Disabled
🐛   tunnel: Disabled:disabled
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [cluster.local] Deploying echo-same-node service...
✨ [cluster.local] Deploying DNS test server configmap...
✨ [cluster.local] Deploying same-node deployment...
✨ [cluster.local] Deploying client deployment...
✨ [cluster.local] Deploying client2 deployment...
✨ [cluster.local] Deploying echo-other-node service...
✨ [cluster.local] Deploying other-node deployment...
✨ [host-netns] Deploying cluster.local daemonset...
✨ [host-netns-non-cilium] Deploying cluster.local daemonset...
✨ [cluster.local] Deploying echo-external-node deployment...
🐛 Validating Deployments...
⌛ [cluster.local] Waiting for deployments [client client2 echo-same-node] to become ready...
⌛ [cluster.local] Waiting for deployments [echo-other-node] to become ready...
⌛ [cluster.local] Waiting for CiliumEndpoint for pod cilium-test/client-6f6788d7cc-dldfl to appear...
⌛ [cluster.local] Waiting for CiliumEndpoint for pod cilium-test/client2-bc59f56d5-p8dqp to appear...
⌛ [cluster.local] Waiting for pod cilium-test/client2-bc59f56d5-p8dqp to reach DNS server on cilium-test/echo-same-node-58f99d79f4-bdp4l pod...
🐛 Error looking up localhost from pod cilium-test/client2-bc59f56d5-p8dqp to server on pod cilium-test/echo-same-node-58f99d79f4-bdp4l: command terminated with exit code 1:
🐛 Error looking up localhost from pod cilium-test/client2-bc59f56d5-p8dqp to server on pod cilium-test/echo-same-node-58f99d79f4-bdp4l: context deadline exceeded:
connectivity test failed: timeout reached waiting lookup for localhost from pod cilium-test/client2-bc59f56d5-p8dqp to server on pod cilium-test/echo-same-node-58f99d79f4-bdp4l to succeed (last error: context deadline exceeded)

General Information

• Cilium CLI version (run cilium version) : v0.14.7 (cilium v1.13.4)
• Orchestration system version in use (e.g. kubectl version, ...) 1.26.5
• Platform / infrastructure information (e.g. AWS / Azure / GCP, image / kernel versions)
• Link to relevant artifacts (policies, deployments scripts, ...)
• Generate and upload a system zip: cilium sysdump

How to reproduce the issue

1. instruction 1
2. instruction 2

[sathieu]

Closing, the problem being the use of default deny policy

[DivyanshuSaxena]

Hi @sathieu If you don't mind sharing, how did you turn off the default deny policy? The official documentation does not make any note of this.

[sathieu]

there is no default deny policy by default unless you have set it.

Either with a CiliumClusterwideNetworkPolicy or with always enforcement mode

(sorry for the delay).