search

RediSearch / RSCoordinator

RediSearch coordinator
Other
24 stars 10 forks source link

Memory leak #262

Open Skoucail opened 3 years ago

Skoucail commented 3 years ago

I'm suspecting a huge memory leak in RSCoordinator or RediSearch. Today I upgraded my clusters to all the latest versions (Redis 6.2.3, RSCoordinator 2.0.5) but I believe the problem is still present.

Basicly our nodes just keep climbing in memory useage untill the server goes OOM and the process redis-server crashes (writing a GDB core file in the proccess) and restarts. Dropping 95% of its memory useage.

On the screenshot below you can see our cluster (in RedisInsight). Depending on when the last crash happend the memory useage is between 835MB and 7.07GB (the nodes have 8GB ram available to use, guessing for 46k keys on each node that should be plenty) After a reboot the nodes start of at around 200MB ram. (this feels to me to be a normal amount for a redis node with 46k keys and some RediSearch indexes in it)

image

ashtul commented 3 years ago

@Skoucail can you please share more details about your schema and hashes? Do you update the hashes?

Skoucail commented 3 years ago

@ashtul We are using multiple schema and some hashes get updated frequently Combined in our schema's we also use probably every available type of field (text, tag, numeric, location)

To improve the performance of our updates we did already activate PARTIAL_INDEXED_DOCS in the configuration. But the memory leak was present even before that change (and also in the previous version)

Even over a timespan of a few (7) day you can see the memory goes up in 1 steady line image