[Feature] Support Cilium - add parsing of conntrack information

[fstr]

Problem Statement

We are using Cilium in kube-proxy-less mode (no kube-proxy). Routing is done via ebpf host routing. This means that the conntrack information is not available any longer in the Kubernetes node's conntrack system.

As a result of our switch to Cilium, network costs in Kubecost are not attributed correctly any longer.

Solution Description

Instead of using conntrack tools, the conntrack information can be listed using cilium bpf ct list global in the Cilium agent, which runs in privilged mode as a DaemonSet on every node. As far as I know, the information is stored on the host in /sys/fs/bpf/tc/globals/cilium_ct4_global.

The information can be accesses programatically. Other vendors like Cast AI already support it in their open source egressd component.

Alternatives

We're using egressd at the moment. While it provides insights into network traffic, the cost component of Kubecost is obviously missing.

Additional Context

• I have reached out to Cilium maintainers and they confirmed that parsing this information would be the way to go
• I have chatted with other people who wrote their own parser for this
• Egressd works
• I reached out to opencost, which informed me that network cost components are part of Kubecost
• I checked the https://github.com/kubecost/conntrack-parser repository but it looks like you took it private, so we can't contribute

Troubleshooting

• [X] I have read and followed the issue guidelines and this is a feature request only for the Kubecost application.
• [X] I have searched other issues in this repository and mine is not recorded.

[chipzoller]

Thanks for the enhancement request here. It seems worthwhile to add specific support for Cilium, perhaps via a container flag which can be configured via a Helm value. The current version of network costs doesn't actually use conntrack any longer and instead requires on sys calls to gather the traffic flows.

Instead of using conntrack tools, the conntrack information can be listed using cilium bpf ct list global in the Cilium agent, which runs in privilged mode as a DaemonSet on every node. As far as I know, the information is stored on the host in /sys/fs/bpf/tc/globals/cilium_ct4_global.

So when it comes to adding support for Cilium in network costs, would its Pods be able to simply parse this file or would they need to possibly exec into Cilium's agent Pods as well (or in place of)? Trying to understand the level of effort required to support Cilium here.

[fstr]

Hi @chipzoller, thanks for your response.

As far as I know, it is possible to programmatically read the ebpf information from any program running on the same host as the Cilium agent. I know that this was done by other people already. An example can be found in the linked project in my original post.

While it's possible to "exec" into the Cilium agent pod, I'd say you want to gather the information directly from the host system.

[Nils98Ar]

@fstr So the network cost feature does not currently work correctly if cilium kubeProxyReplacement is enabled?

[fstr]

@Nils98Ar correct, only pods that use hostNetwork (e. g. cilium-agent) will be tracked, which causes mis-attribution. All other pod traffic won't be tracked at all.

[Nils98Ar]

@fstr @chipzoller Using the Kubernetes node's traffic flow information also does not work with the Cilium default routing mode encapsulation/tunnel (vxlan/geneve) regardless of wether you run Cilium kube-proxy-less or not. I think Ciliums conntrack information needs to be used here as well.

When pod p1 on node 1 communicates with a pod p2 on node 2 you only see a connection from pod p1 IP tonode 2 IP ("remote node") in the traffic flows (at least in the conntrack table) on node 1. You do not see the IP pod p2 there.

So you cannot use the traffic classification by destination (internet, in-zone, in-region, etc.) for Pod-IPs within the Cluster. E.g. we wanted to add the Ingress Controllers Pod IPs to the internet destinations.

[chipzoller]

Hello, in an effort to consolidate our bug and feature request tracking, we are deprecating using GitHub to track tickets. If this issue is still outstanding and you have not done so already, please raise a request at https://support.kubecost.com/.