Investigate metrics across kube-proxy, ipvs, iptables rules.

[jayunit100]

ill leave some notes in here on how to do this.

1) the tutorial for prometheus metrics in this repo shows how to create prometheus metrics on a kind cluster

2) Update kube-proxy daemonset to publish metrics somewhere easy to access ? for some reason on VMC 0.0.0.0:10249 isnt the curlable IP.

3) Run node-exporter also so that you can export iptables and other linux time series metrics... https://github.com/prometheus/node_exporter/issues/637

4) See what happens to those metrics as you add 1,10,100 pods to the system, graph them.

options:

5.0) Antrea and calico both export metrics - combining those also would be interesting
5.1) Look at kube-proxy code, find metrics that might be good to export and PR them upstream

any interesting patterns ?

[jayunit100]

copy paste (kubernetes/pkg/proxy)

        // EndpointChangesTotal is the number of endpoint changes that the proxy
        // has seen.
        EndpointChangesTotal = metrics.NewCounter(
                &metrics.CounterOpts{
                        Subsystem:      kubeProxySubsystem,
                        Name:           "sync_proxy_rules_endpoint_changes_total",
                        Help:           "Cumulative proxy rules Endpoint changes",
                        StabilityLevel: metrics.ALPHA,
                },
        )

somewhere interesting worth counting, like ...

// detectStaleConnections modifies <staleEndpoints> and <staleServices> with detected stale connections. <staleServiceNames>
// is used to store stale udp service in order to clear udp conntrack later.
func detectStaleConnections(oldEndpointsMap, newEndpointsMap EndpointsMap, staleEndpoints *[]ServiceEndpoint, staleServiceNames *[]ServicePortName) {
        for svcPortName, epList := range oldEndpointsMap {
                if svcPortName.Protocol != v1.ProtocolUDP {
                        continue
                }

might benefit from having a new metric to count how many of these are occuring.

[jayunit100]

(possibly if you want to be fancy, use the prometheus-operator exposed over a hostPort on your kubeadm or capi clusters)

[jayunit100]

╰─➤  curl localhost:8001/api/v1/nodes/antrea-2-md-0-65ccd96655-gqst2/proxy/metrics/cadvisor