Memory Leak with aot package ListDeployments and ListStatefulSet

[guillaume-chervet]

Describe the bug SlimFaas is calling ListDeployments and ListStatefulSet every Second (in the futur we will use watch). I think of memory leak because memory increase without ending. Previous version worked on production without any crash with 180Mo of RAM set and the same code. The only change is the switch to AOT packages.

If I remove calls, the memory keep very stable.

https://github.com/AxaFrance/SlimFaas/blob/main/src/SlimFaas/ReplicasSynchronizationWorker.cs https://github.com/AxaFrance/SlimFaas/blob/main/src/SlimFaas/Kubernetes/KubernetesService.cs

All RAM goes down in the graph bellow are OOM Killed.

Kubernetes C# SDK Client Version "KubernetesClient.Aot" Version="13.0.12"

Dotnet Runtime Version .NET 8

[WeihanLi]

Is it possible to analyze some memory info from a dump?

Think the HttpRequestMessage/HttpResponseMessage and the HttpContent should be disposed, maybe we could try to add using for them firstly

[guillaume-chervet]

Hi fixed it in my side by this commit https://github.com/AxaFrance/SlimFaas/commit/d71b2ba065f416ae22bd0340407c928e9226e051

But it is still a bug it seem dispose does not dispose well.

I will try to memory dump.

[tg123]

yes dump please i will try to repo by calling list

[guillaume-chervet]

I had a add week, i will try to do it next week!

[tg123]

i ran the load test last weekend for whole afternoon, but did not see any leak. code is simple

for(;;) { listdeployment; sleep 5s}

[tg123]

my test code

using k8s;

var config = KubernetesClientConfiguration.BuildDefaultConfig();
IKubernetes client = new Kubernetes(config);
Console.WriteLine("Starting Request!");

for (; ;){
    var list = client.AppsV1.ListDeploymentForAllNamespaces();
    foreach (var item in list.Items)
    {
        Console.WriteLine(item.Metadata.Name);
    }

    Thread.Sleep(5000);
    GC.Collect();
}

[guillaume-chervet]

Hi @tg123 @WeihanLi

I was creating disposing a new Kubernetes client at each iteration.

[guillaume-chervet]

I still have now a tigth memory leak. 10 Mo in 2 productions week. But i have to investigate to know the source.

[tg123]

[guillaume-chervet]

hi @tg123 , I still have the problem with the latest verison by creating a new kubernetes client every time =>

with IKubernetes client = new Kubernetes(config);

client.Dispose();

every seconds

[tg123]

@guillaume-chervet could you please do a mem dump so we can debug where is leaking

[guillaume-chervet]

I will as soon as i can @tg123 very sorry for the delay

[guillaume-chervet]

Thank you very much for your help and answers @tg123