search

nats-io / nats-streaming-server

NATS Streaming System Server
https://nats.io
Apache License 2.0
2.51k stars 283 forks source link

Possible memory leak in v0.21.1 #1176

Closed hullarb closed 3 years ago

hullarb commented 3 years ago

Hi,

First of all let me thank you, we've been using nats streaming for almost 3 years and been quite satisfied with it. Sorry to see the deprecation notice.

We set up NATS streaming in clustering mode with file store. After upgrading our dev system from 0.20.0 to 0.21.1 we seen strange memory growth, which looks like leaking memory. We have running 3 replicas on k8s and with the earlier versions memory usage was quite stable for us (in the current production cluster it's almost a straight horizontal line around ~100 Mb for 30 days back). After upgrading to version 0.21.1 memory usage constantly increasing until we let it, without memory limits it went up to several GB or with the limit it gets OOM killed regularly.

if i run the basic clustering test with increasing the number of messages to a big number and adding one millisec delay between them: https://github.com/nats-io/nats-streaming-server/blob/master/server/clustering_test.go#L709 triggers 4x larger memory consumption on tag v0.21.1 than on v0.20.0

kkoppel commented 3 years ago

I am seeing the same issue with memory usage growing forever after switching our cluster to v0.21.1, while it was fine previously on v0.19.0.

kozlovic commented 3 years ago

@hullarb @kkoppel Thank you for the report. I think I know what is happening. Will investigate more and if my suspicion is correct, will have a fix soon.

kozlovic commented 3 years ago

@hullarb @kkoppel Would you be able to test from the branch the PR #1177 is based off to see if that was the issue? That would be appreciated. Thanks!

hullarb commented 3 years ago

Thank you @kozlovic for the quick response and fix! i've deployed this version few hours before in a dev cluster and so far memory usage looks stable and actually quite low. It looks to me working.

ghost commented 3 years ago

Hi @kozlovic, do you have any plans to issue a patched version of nats-streaming?

kozlovic commented 3 years ago

@sergeiepam Yes, there will be a 0.21.2 release. When, as soon as possible, but can't give you exact date. Maybe early next week.

kkoppel commented 3 years ago

I can also confirm that after 12h of running on the fixed version memory usage is stable.

kozlovic commented 3 years ago

@hullarb @kkoppel @sergeiepam I have release v0.21.2: https://github.com/nats-io/nats-streaming-server/releases/tag/v0.21.2

Note that the docker image may not be available until sometime tomorrow.

Thanks!