Reset connection to NodePort if pod is on the same node

senelschikov commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Version

equal or higher than v1.16.0 and lower than v1.17.0

What happened?

When I try to connect to node with service NodePort from outside the k8s cluster and a target pod on the same node, connection will by reset by the client. However, if I try to connect to another node where the target pod is not running, it will succeed. Connection from any node or pod of the k8s cluster is successful as well. HostPort works also.

How can we reproduce the issue?

Install cilium with helm
Helm flags are: upgrade cilium cilium/cilium --version 1.16.1 --set devices=br1,br0 --set direct-routing-device=br1 --namespace=kube-system --set upgradeCompatibility=1.15 -f old-values-1.15.yaml

old-values-1.15.yaml:

cluster:
  name: kubernetes
hubble:
  enabled: true
  metrics:
    enabled:
    - dns:query;ignoreAAAA
    - drop
    - tcp
    - flow
    - icmp
    - http
  relay:
    enabled: true
  ui:
    enabled: false
ipam:
  operator:
    clusterPoolIPv4PodCIDRList: 192.168.0.0/16
k8sServiceHost: api.k8s.dev
k8sServicePort: 6443
kubeProxyReplacement: true
operator:
  replicas: 1
serviceAccounts:
  cilium:
    name: cilium
  operator:
    name: cilium-operator
tunnel: vxlan
upgradeCompatibility: "1.14"

Cilium Version

cilium-cli: v0.15.8 compiled with go1.21.0 on linux/amd64 cilium image (default): v1.14.1 cilium image (stable): v1.16.1 cilium image (running): 1.16.1

Kernel Version

Linux hel379 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux Linux hel439 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

Kubernetes Version

Client Version: v1.28.7 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.7

Regression

No response

Sysdump

No response

Relevant log output

From outside
kubectl -n kube-system exec cilium-nlgjf -c cilium-agent -- cilium monitor --related-to 738
Listening for events on 48 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
-> endpoint 738 flow 0xeaaa0de9 , identity world->1861 state new ifindex lxcd67573e25793 orig-ip 62.71.111.97: 62.71.111.97:57938 -> 192.168.9.207:80 tcp SYN
-> stack flow 0xafdb0a8a , identity 1861->world state reply ifindex 0 orig-ip 0.0.0.0: 192.168.9.207:80 -> 62.71.111.97:57938 tcp SYN, ACK
time="2024-09-17T11:37:24Z" level=info msg="Initializing dissection cache..." subsys=monitor
-> endpoint 738 flow 0xeaaa0de9 , identity world->1861 state established ifindex lxcd67573e25793 orig-ip 62.71.111.97: 62.71.111.97:57938 -> 192.168.9.207:80 tcp ACK
-> endpoint 738 flow 0xeaaa0de9 , identity world->1861 state established ifindex lxcd67573e25793 orig-ip 62.71.111.97: 62.71.111.97:57938 -> 192.168.9.207:80 tcp ACK
-> endpoint 738 flow 0xbf7485fa , identity world->1861 state established ifindex lxcd67573e25793 orig-ip 62.71.111.97: 62.71.111.97:57938 -> 192.168.9.207:80 tcp RST
-> endpoint 738 flow 0xeaaa0de9 , identity world->1861 state established ifindex lxcd67573e25793 orig-ip 62.71.111.97: 62.71.111.97:57938 -> 192.168.9.207:80 tcp ACK
-> stack flow 0x0 , identity 1861->world state reply ifindex 0 orig-ip 0.0.0.0: 192.168.9.207:80 -> 62.71.111.97:57938 tcp RST

From node with running pod 
# kubectl -n kube-system exec cilium-nlgjf -c cilium-agent -- cilium monitor --related-to 738
Listening for events on 48 CPUs with 64x4096 of shared memory
Press Ctrl-C to quit
time="2024-09-17T11:38:38Z" level=info msg="Initializing dissection cache..." subsys=monitor
-> endpoint 738 flow 0x9b1ae3a2 , identity host->1861 state new ifindex lxcd67573e25793 orig-ip 192.168.9.145: 192.168.9.145:44008 -> 192.168.9.207:80 tcp SYN
-> stack flow 0x99cfd41d , identity 1861->host state reply ifindex 0 orig-ip 0.0.0.0: 192.168.9.207:80 -> 192.168.9.145:44008 tcp SYN, ACK
-> endpoint 738 flow 0x9b1ae3a2 , identity host->1861 state established ifindex lxcd67573e25793 orig-ip 192.168.9.145: 192.168.9.145:44008 -> 192.168.9.207:80 tcp ACK
-> endpoint 738 flow 0x9b1ae3a2 , identity host->1861 state established ifindex lxcd67573e25793 orig-ip 192.168.9.145: 192.168.9.145:44008 -> 192.168.9.207:80 tcp ACK
-> stack flow 0x99cfd41d , identity 1861->host state reply ifindex 0 orig-ip 0.0.0.0: 192.168.9.207:80 -> 192.168.9.145:44008 tcp ACK, FIN
-> endpoint 738 flow 0x9b1ae3a2 , identity host->1861 state established ifindex lxcd67573e25793 orig-ip 192.168.9.145: 192.168.9.145:44008 -> 192.168.9.207:80 tcp ACK, FIN

Anything else?

cilium-config.txt

Cilium Users Document

[ ] Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

[X] I agree to follow this project's Code of Conduct

youngnick commented 3 weeks ago

This sounds similar to #24318. Could you see if setting endpointRoutes.enabled in Helm helps?

uh-cookie commented 1 week ago

@youngnick We are experiencing a very similar issue but on AWS AL2023.

I've attached pwru output from the offending node pwru-out.txt and the Cilium configuration: cilium-config.txt (30080 is NodePort, 9200 is the container port, 10.2.3.234 is instance without Cilium).

If we downgrade back to 1.15.7 the issue disappears.

If there is anything else please let me know - happy to provide more information.

Cilium Version Client: 1.16.2 3261be3f 2024-09-20T08:39:29+00:00 go version go1.22.7 linux/amd64 Daemon: 1.16.2 3261be3f 2024-09-20T08:39:29+00:00 go version go1.22.7 linux/amd64

Kernel Version Linux 6.1.106-116.188.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Aug 27 07:00:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version Server Version: v1.30.0

cilium / cilium