Retina's networkobservability_tcp_connection_remote metric does not correlate with the number of SNAT ports used. In the case of SNAT ports exhaustion, we are seeing few outgoing connections from the nodes according to Retina.
Currently in Retina, we are using netstat as the way to get the metric, which we have identified as the reason why there was a discrepancy to begin with. This is because each pod in a Kubernetes cluster has its own network namespace. The network namespace is a feature of the Linux kernel that isolates the network stack of processes from each other. This means that each pod has its own network interfaces, IP addresses, routing tables, and so on. So, when we run netstat on the node, we are looking at the network connections in the network namespace of the node itself, not the network namespaces of the pods running on that node. If we go into a pod's network namespace and run netstat, we will now be able to see the network connections within that specific pod's network namespace. This behavior is the same for netstat on both Windows and Linux.
conntrack is a core feature of Netfilter that allows the kernel to keep track of all logical network connections or sessions, and thereby relate all the packets which make up each connection. Netfilter maintains a connection tracking table that stores information about all ongoing connections. Even though each pod in Kubernetes has its own network namespace, the connection tracking table is global to the entire system. This means that conntrack can see connections made from all network namespaces, including those of individual pods.
networkobservability_tcp_connection_remotemetric does not correlate with the number of SNAT ports used. In the case of SNAT ports exhaustion, we are seeing few outgoing connections from the nodes according to Retina.netstatas the way to get the metric, which we have identified as the reason why there was a discrepancy to begin with. This is because each pod in a Kubernetes cluster has its own network namespace. The network namespace is a feature of the Linux kernel that isolates the network stack of processes from each other. This means that each pod has its own network interfaces, IP addresses, routing tables, and so on. So, when we runnetstaton the node, we are looking at the network connections in the network namespace of the node itself, not the network namespaces of the pods running on that node. If we go into a pod's network namespace and runnetstat, we will now be able to see the network connections within that specific pod's network namespace. This behavior is the same fornetstaton both Windows and Linux.conntrackis a core feature of Netfilter that allows the kernel to keep track of all logical network connections or sessions, and thereby relate all the packets which make up each connection. Netfilter maintains a connection tracking table that stores information about all ongoing connections. Even though each pod in Kubernetes has its own network namespace, the connection tracking table is global to the entire system. This means thatconntrackcan see connections made from all network namespaces, including those of individual pods.