Closed kurtmc closed 3 years ago
I believe this is fixed with https://github.com/TheThingsNetwork/lorawan-stack/pull/3704 but it isn't part of a 3.10 release yet. We're releasing 3.10.10 today.
@rvolosatovs I triaged the issue but if you feel confident that this is resolved, please close.
We are still seeing this in 3.10.10
@adriansmares do you have any input here based on your research of go-redis
library bugs? Can this be related?
@adriansmares do you have any input here based on your research of
go-redis
library bugs? Can this be related?
I've posted my findings here.
TLDR: Could be, but I cannot tell. The connection reaper bug could be old, and cause this issue (the one we're in right now), or it could be new and introduced as part of the v8.4.1
release, and then it's probably unrelated.
Possibly ref https://github.com/go-redis/redis/issues/1657
Looks like this is resolved in v3.11.2, we have been running for about 6 days and have not run into this redis issue.
Summary
After some time running lorawan we will see that the network traffic to our redis instance drops off and a large number of errors being reported by the lorawan application. Restarting the lorawan application resolves the issue.
Steps to Reproduce
We can only reproduce this issue in our production environment, so I suspect you need to have a lot of traffic on lorawan to run into this issue.
lorawan-config.txt
What do you see now?
What do you want to see instead?
...
Environment
How do you propose to implement this?
I think there is a leak related to the redis connection pool. I am unsure how to fix it.
How do you propose to test this?
...
Can you do this yourself and submit a Pull Request?
...