Memory and CPU leak in agent and service

[chriswessels]

Hey there,

I'd been running release sha-54d4905 on Kubernetes for a few days before upgrading to a new release, and noticed a huge drop in memory usage for the indexer-agent and indexer-serivce.

Looking back at memory usage for the deployments:

Both seem to have a fairly consistent and linear growth in memory usage of time.

Let me know if there are any further details I can provide!

Chris

[chriswessels]

Looks like it's leaking CPU cycles too:

[chriswessels]

Looks like indexer-agent and indexer-service still have a memory leak problem. I've been using a Kubernetes CronJob to automatically restart them every day to keep memory usage under control. Here's a gist for anyone that wants to do the same: https://gist.github.com/chriswessels/8271f82a0ae7342d7d0822ea1e796246

[aasseman]

I'm also observing this issue on v0.18.6. However I've found a good way to accelerate it: send lots of GraphQL queries to the indexer-agent management endpoint. I've also noticed that the queries would take longer to execute as memory usage grows.

However I am not sure if certain types of queries are worse than others. Read below.

Context As @fordN and @chriswessels already know, I'm currently running experiments that require constantly updating the Agora models. In this particular case, I am updating the model variables for each subgraph every 3 minutes. In the plot below, every drop is the OOM killer killing the indexer-agent. The frequency increased around May 10th, when I deployed my experiment to the indexer.

[aasseman]

Update: The problem is still present on v0.19.2.

[kaiwetlesen]

Update: immediately preceding the OOM crash, queries begin failing. This causes a drop in quality of service.

[aasseman]

Update: the problem is still present on v0.20.12

[alex-pakalniskis]

Figment appears to be experiencing this issue too (shared with me on March 14, 2024)