search

RedisGrafana / grafana-redis-datasource

Redis Data Source for @Grafana allows connecting to any @Redis database On-Premises and in the Cloud.
https://redisgrafana.github.io
Apache License 2.0
148 stars 39 forks source link

Experiencing memory leak in Grafana docker seemingly stemming from this plugin #116

Closed emmanuelnk closed 3 years ago

emmanuelnk commented 3 years ago

Hi I have Grafana running in a docker container (grafana/grafana) in ECS Fargate Service on AWS and I have been getting memory leaks when I (incorrectly?) enter custom redis commands in grafana.

Context

When I enter a custom command in redis timeseries datasource, there seems to be a memory leak that occurs immediately afterwards. Memory consumption goes from 20% to 95% in a few minutes while spitting out this log message. This is the log from ECS the moment before and while it happens:

t=2020-12-28T07:07:26+0000 
lvl=eror 
msg=PANIC 
logger=plugins.backend 
pluginId=redis-datasource 
command="should never get here" 
query="TS.RANGE stream:9DoF_0001:* 1609139215082 1609139245082"

I traced the error output it to this line of code: https://github.com/RedisGrafana/grafana-redis-datasource/blob/cb60f57b2058ea7ee65d8c4de723bbcb54a27cb2/pkg/redis-custom.go#L38

Is there any insight in what might be going on here? I may have written a bad query to redis timeseries (the query obviously doesn't accept wildcards) however I don't think the result should be such a severe memory leak... so I thought I'd bring up this issue.

mikhail-vl commented 3 years ago

@emmanuelnk Thank you for submitting the issue. I will take a look at it.

You are correct, TS.RANGE command requires the key and does not support wildcards. To get time-series from multiple keys you should use TS.MRANGE command and query using labels.

If the command can't be executed properly, it's PANIC in Radix golang client. I will check with the developer on how I can improve it and get back to you.

mikhail-vl commented 3 years ago

PANIC fixed in Radix 3.7.0: https://github.com/mediocregopher/radix/issues/250

mikhail-vl commented 3 years ago

@emmanuelnk We were not able to reproduce memory leaks using docker containers, we have not tried RCS Fargate. The PANIC was fixed in the Radix client and now you should see the proper error coming from Redis in the data source.

The fix is a part of the upcoming release 1.3.1, which you can BUILD yourself following BUILD instructions to test. Please let me know if you have any questions.