microsoft / reverse-proxy

A toolkit for developing high-performance HTTP reverse proxy applications.
https://microsoft.github.io/reverse-proxy
MIT License
8.43k stars 828 forks source link

Memory leak with KubernetesClient #2517

Open dpetrovych opened 3 months ago

dpetrovych commented 3 months ago

While running YARP as ingress controller we see overall raising pattern of memory consumption and frequent restarts due to hitting pod limits.

To Reproduce

Run release/2.1/samples/KubernetesIngress.Sample/Ingress sample ingress for a breif while (10-15 minutes). Get a GC Dump.

image

Issue has a signature / related to: https://github.com/kubernetes-client/csharp/issues/1123 Present in KubernetesClient.Basic v.9.0.28. Fixed in KubernetesClient.Basic v.10.0.1

Further technical details

dpetrovych commented 3 months ago

I started using YARP v.2.2.0-preview.1 (where KubernetesClient is update to v.10) and pattern of GC dumps have changed. Problem is not fully resolved, service continues to increse RAM usage daily. Here is comparison though 26 hours: one of the most numerable objects is CancelationTokenSource.

image

Also not sure, if such report would indicate a cyclic dependecy that prevents garbage collection: image

MihaZupan commented 2 months ago

Thanks for raising the issue. It'd make sense for us to move to the more recent KubernetesClient version with the fix.