ops-view causing kube-api to be unavailable from node

[integrii]

Reproduce:

• Deploy ops-view on a cluster with hundreds of nodes
• Up resources for redis and ops-view so that it does not crash
• Start checking that your node can communicate with the kubernetes API
• Wait 24 hours

Results: You will notice that there are various 20 minute periods of time where the node running ops-view can not communicate with the cluster ip of the api kube-api service. This effects CNI IP provisioning, pod creation, pod termination, and anything else that requires the API. The issue resolves after a period of time and beings working. Other nodes are not affected by this connection blocking.

Assumptions: Something that ops-view does is overwhelming the API, causing the API to stop listening to requests from ops-view.

Notes: We have reproduced this on five different clusters. The error only came to light because we monitor our clusters with Kuberhealthy, which runs synthetic tests to ensure all pod creation and removal is working in a timely fashion.

[hjacobs]

@integrii what resource limits to you give the API server? Both the API server resources or etcd might be the bottle neck. kube-ops-view currently polls all pods/nodes periodically (no watch).

[integrii]

The API server is the default that kops provisions, which I think is the default of Kubernetes.

How much work is it to re-implement your dashboard with a watch? We absolutely love the display and were running it on the cafe screens for a couple days. I am sad that I can't run this on our large production clusters.

[geNAZt]

We ran into a similar problem with ops-view. We noticed that it uses a huge amount of tcp_mem leading me to believe it somehow leaks either API connections or websocket connections for the UI. Once we uninstalled ops-view our cluster got back to normal operation.

[zerkms]

Another case here as well (the same as @geNAZt): after leaving kube-ops-view running during weekend I returned to the office to find the node that hosts the ops-view application to be barely responsible with tcp_mem of ~249k

[zerkms]

That's a 20 hours overview of the sockstat (the mem metric).

The step down at the beginning is a kube-ops-view app restart.

[hjacobs]

@zerkms thanks for the insights, I'll try to reproduce the behavior.

[zerkms]

Another important detail: that leak happens regardless, but the rate of leaking is significantly higher if the app and redis run on different nodes.

So for myself I used pod affinity to run them together and created a cronjob that restarts the app every several hours :man_shrugging:

[hjacobs]

@zerkms does it only happen when using Redis? (kube-ops-view also works without Redis)

[zerkms]

Oh, I did not know that. I'll check that tomorrow, but from my other observations the answer 99% likely would be "yes".

[hjacobs]

I also have a demo setup deployed on https://kube-ops-view.demo.j-serv.de/ (it uses Redis), I did not observe the memory leak there so far, but I also don't have a browser window open permanently :smirk:

[zerkms]

I did not observe the memory leak

How did you check that? It's not RSS leak, it's a kernel tcp sockets memory leak

Check cat /proc/net/sockstat: the mem field is the one of interest.

[hjacobs]

@zerkms you are right, what I wanted to say is that I did not observe the symptom/impact of any memory leak (unresponsive node):

cat /proc/net/sockstat | grep TCP.*mem
TCP: inuse 49 orphan 0 tw 33 alloc 484 mem 29990

[zerkms]

I did not observe the symptom/impact of any memory leak (unresponsive node):

Check cat /proc/sys/net/ipv4/tcp_mem

When the value in sockstat reaches the number in 3rd column the kernel would experience problems creating new or using the current network connections.

In our case it takes about 3 days to get an almost unresponsive node.

[zerkms]

@hjacobs breaking news: that leak is only happening when there are browser clients open with the application.

And another important note: when there are clients showing the page - the rate of leaking is lower when they (redis + the app) are on the same node, and higher when on different. (it leaks nevertheless, but it's a curious observation as well)

[hjacobs]

@zerkms thanks for all your investigations --- kube-ops-view is definitely causing high TCP mem as I also can see on my K3s demo:

root@k3s-demo:~# cat /proc/sys/net/ipv4/tcp_mem
22200   29602   44400
root@k3s-demo:~# cat /proc/net/sockstat | grep TCP.*mem
TCP: inuse 50 orphan 0 tw 34 alloc 492 mem 30335
root@k3s-demo:~# cat /proc/sys/net/ipv4/tcp_mem
22200   29602   44400
root@k3s-demo:~# kubectl get pod
NAME                                    READY   STATUS    RESTARTS   AGE
kube-ops-view-7b9dd46fd8-vfqgt          1/1     Running   0          12d
kube-ops-view-redis-577f846477-lrgk2    1/1     Running   0          25d
kube-resource-report-5f77c8f5d9-lwtnl   2/2     Running   0          10d
kube-web-view-6f8d9ff748-xcp5c          1/1     Running   0          35h
nginx-c9767ffdf-22tgk                   1/1     Running   0          25d
nginx-c9767ffdf-8qzqr                   1/1     Running   0          25d
root@k3s-demo:~# kubectl delete pod kube-ops-view-7b9dd46fd8-vfqgt
pod "kube-ops-view-7b9dd46fd8-vfqgt" deleted
root@k3s-demo:~# cat /proc/net/sockstat | grep TCP.*mem
TCP: inuse 48 orphan 1 tw 26 alloc 173 mem 3

=> TCP mem goes from 30335 Kernel pages (?) to 3 Kernel pages. Not sure what the unit is but this suggests "Kernel pages" (4096 Bytes).

[hjacobs]

Could reproduce it again:

cat /proc/net/sockstat | grep TCP.*mem
TCP: inuse 48 orphan 0 tw 28 alloc 697 mem 44386
...
kubectl delete pod kube-ops-view-7b9dd46fd8-g544t
...
cat /proc/net/sockstat | grep TCP.*mem
TCP: inuse 53 orphan 3 tw 13 alloc 212 mem 38
 pod kube-ops-view-7b9dd46fd8-g544t

[zerkms]

Did you have it opened in browser?

[integrii]

Bump. I saw this software on a mailing list and remembered how cool it is. Any progress on reduction of client connection spam to the API server yet? I think refactoring to a reflector that watches the API could solve the problems here.