[Question]: Need help understanding the source of a connection that is being dropped because of network policy

[viveksinghggits]

Hey Folks, I was not able to figure out the proper channel to ask the question, that's why reaching out here. If this is the not the right place, please feel free to let me know. So, in the logs from pods -l app=ovnkube-node from namespace openshift-ovn-kubernetes we can see that a pod is trying to communicate with kubernetes api server

2024-03-12T12:26:48.710Z|3252252|acl_log(ovn_pinctrl0)|INFO|name="mongodb-logical_egressDefaultDeny", verdict=drop, severity=alert, direction=from-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:80:02:e9,dl_dst=0a:58:0a:80:02:01,nw_src=a.b.c.233,nw_dst=a.0.b.45,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=44064,tp_dst=6443,tcp_flags=syn

We see a pod in the concerned namespace with IP a.b.c.233 but we don't think (kind of sure) that the commands used in this pod are making any connections with kube-apisever. And I am not able to find out which exact process from that pod is trying to make connection with the api-server. I tried to exec into the pod to see if which process is running on port 44064 (netstat) but couldn't see a process on that port. Am I understanding the log incorrectly here or there is actually a process in the pod that is making a connection with kue-apiserver? If there are some other ways to figure that out, I would appreciate that.

[tssurya]

Hi @viveksinghggits !

Right so the logging indicates there is a default deny ACL for egress traffic in the namespace where the pod with IP a.b.c.233:44064 lives and so a syn packet going from this podIP towards a.0.b.45:6443 is getting dropped.

So it mostly has to be a containerPort or one of the port types defined on the podSpec for one of the containers.

[viveksinghggits]

@tssurya Right, I understand. But the problem is the pod spec does't actually have a port specified and the reason is, it's not a long living pod we are not running any service (that needs to be exposed) in the pod. We create it to just run two commands. And then the pod completes/terminates. None of those two commands communicate with kueb-apiserver. And that is what's puzzling me.

2024-03-12T20:58:09.469Z|00575|acl_log(ovn_pinctrl0)|INFO|name="mongodb-logical_egressDefaultDeny", verdict=drop, severity=alert, direction=from-lport: tcp,vlan_tci=0x0000,dl_src=0a:58:0a:83:00:70,dl_dst=0a:58:0a:83:00:01,nw_src=a.b.c.112,nw_dst=d.e.f.42,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=49562,tp_dst=6443,tcp_flags=syn

k get pods -n mongodb-logical kanister-job-wtkfb -owide
k eNAME                 READY   STATUS    RESTARTS   AGE   IP             NODE                                        NOMINATED NODE   READINESS GATES
kanister-job-wtkfb   1/1     Running   0          35s   a.b.c.112   ip-defg.zone.compute.internal   <none>           <none>

k exec -it -n mongodb-logical kanister-job-wtkfb -- bash
root@kanister-job-wtkfb:/# netstat -tulpn | grep :49562
root@kanister-job-wtkfb:/# cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                     

And podspec is here that doesn't actually have the port specified.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[viveksinghggits]

After further investigation it does look like the application that we were running in the pod tried to talk to apiserver because we were importing a package and that package had init function which established a call to apiServer.