search

cilium / cilium-cli

CLI to install, manage & troubleshoot Kubernetes clusters running Cilium
https://cilium.io
Apache License 2.0
417 stars 210 forks source link

Connectivity tests fail with `--enable-well-known-identities=true`; incorrect policy drops for kube-dns #1674

Open christarazi opened 1 year ago

christarazi commented 1 year ago

Bug report

$ cilium connectivity test --flow-validation=disabled -p
...
ℹī¸  Skipping IPCache check
🔭 Enabling Hubble telescope...
ℹī¸  Hubble is OK, flows: 2262/8190
ℹī¸  Cilium version: 1.14.0
🏃 Running tests...
[=] Test [no-policies]
........................
[=] Test [no-policies-extra]
........
[=] Test [allow-all-except-world]
.......
  ℹī¸  📜 Applying CiliumNetworkPolicy 'allow-all-except-world' to namespace 'cilium-test'..
  [-] Scenario [allow-all-except-world/pod-to-pod]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-0: cilium-test/client-6965d549d5-hpkqt (10.244.1.228) -> cilium-test/echo-other-node-f57db5457-9q9km (10.244.0.47:8080)]
  📄 Following flows...
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-1: cilium-test/client-6965d549d5-hpkqt (10.244.1.228) -> cilium-test/echo-same-node-799c9b99f-xrz4t (10.244.1.116:8080)]
  📄 Following flows...
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-2: cilium-test/client2-76f4d7c5bc-dnj57 (10.244.1.23) -> cilium-test/echo-other-node-f57db5457-9q9km (10.244.0.47:8080)]
  📄 Following flows...
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-3: cilium-test/client2-76f4d7c5bc-dnj57 (10.244.1.23) -> cilium-test/echo-same-node-799c9b99f-xrz4t (10.244.1.116:8080)]
  📄 Following flows...
  [-] Scenario [allow-all-except-world/client-to-client]
  [.] Action [allow-all-except-world/client-to-client/ping-ipv4-0: cilium-test/client-6965d549d5-hpkqt (10.244.1.228) -> cilium-test/client2-76f4d7c5bc-dnj57 (10.244.1.23:0)]
  📄 Following flows...
  [.] Action [allow-all-except-world/client-to-client/ping-ipv4-1: cilium-test/client2-76f4d7c5bc-dnj57 (10.244.1.23) -> cilium-test/client-6965d549d5-hpkqt (10.244.1.228:0)]
  📄 Following flows...
  [-] Scenario [allow-all-except-world/pod-to-service]
  [.] Action [allow-all-except-world/pod-to-service/curl-0: cilium-test/client-6965d549d5-hpkqt (10.244.1.228) -> cilium-test/echo-other-node (echo-other-node:8080)]
  📄 Following flows...
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://echo-other-node:8080" failed: command terminated with exit code 28
  ℹī¸  curl output:

  📄 Flow logs for peer cilium-test/client-6965d549d5-hpkqt:
  ❓ [0] May 31 04:58:52.009: cilium-test/client-6965d549d5-hpkqt -> cilium-test/client2-76f4d7c5bc-dnj57 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (ICMPv4 EchoReply)
  ❓ [1] May 31 04:58:52.009: cilium-test/client-6965d549d5-hpkqt -> cilium-test/client2-76f4d7c5bc-dnj57 to-endpoint FORWARDED EGRESS DROP_REASON_UNKNOWN (ICMPv4 EchoReply)
  ❓ [2] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [3] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [4] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [5] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [6] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [7] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [8] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [9] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [10] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [11] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [12] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [13] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)

  📄 Flow logs for peer cilium-test/echo-other-node:
  ❓ [0] May 31 04:58:52.009: cilium-test/client-6965d549d5-hpkqt -> cilium-test/client2-76f4d7c5bc-dnj57 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (ICMPv4 EchoReply)
  ❓ [1] May 31 04:58:52.009: cilium-test/client-6965d549d5-hpkqt -> cilium-test/client2-76f4d7c5bc-dnj57 to-endpoint FORWARDED EGRESS DROP_REASON_UNKNOWN (ICMPv4 EchoReply)
  ❓ [2] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [3] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [4] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [5] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [6] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [7] May 31 04:58:52.077: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [8] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [9] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [10] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [11] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/kube-dns:53 from-endpoint FORWARDED TRAFFIC_DIRECTION_UNKNOWN DROP_REASON_UNKNOWN (UDP)
  ❓ [12] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 policy-verdict:none EGRESS DROPPED EGRESS POLICY_DENIED (UDP)
  ❓ [13] May 31 04:58:54.578: cilium-test/client-6965d549d5-hpkqt:46559 -> kube-system/coredns-5d78c9869d-hgq6f:53 Policy denied DROPPED EGRESS POLICY_DENIED (UDP)

  Pausing after action failure, press the Enter key to continue:

  [.] Action [allow-all-except-world/pod-to-service/curl-1: cilium-test/client-6965d549d5-hpkqt (10.244.1.228) -> cilium-test/echo-same-node (echo-same-node:8080)]
  📄 Following flows...
^CInterrupt received, cancelling tests...
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://echo-same-node:8080" failed: context canceled
  ℹī¸  curl output:

  📄 No flows recorded for peer cilium-test/client-6965d549d5-hpkqt during action curl-1
  📄 No flows recorded for peer cilium-test/echo-same-node during action curl-1
  Pausing after action failure, press the Enter key to continue:

  [.] Action [allow-all-except-world/pod-to-service/curl-2: cilium-test/client2-76f4d7c5bc-dnj57 (10.244.1.23) -> cilium-test/echo-other-node (echo-other-node:8080)]
  📄 Following flows...
  đŸŸĨ Skipping command execution: context canceled
  ℹī¸  📜 Deleting CiliumNetworkPolicy 'allow-all-except-world' from namespace 'cilium-test'..
^Cconnectivity test failed: context canceled

General Information

How to reproduce the issue

  1. Deploy Cilium with enable-well-known-identities: true
  2. Run cilium connectivity test
github-actions[bot] commented 1 day ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.