[kube-prometheus-stack-36.2.0] Grafana Image render not working

[diegopedroso]

Describe the bug a clear and concise description of what the bug is.

Previously Grafana's versions had a feature that able users to send images/charts embedded in the alert messages. The configuration was something like that:

Recently, the feature was removed and replaced by a custom plugin, to render the images in the same way:

https://grafana.com/grafana/plugins/grafana-image-renderer/

grafana-cli plugins install grafana-image-renderer

Unfortunately, we've had issues with the plugin, it seems some versions (we tested with 9.0.1 - 9.4.2), and we got the errors below:

logger=plugin.manager t=2023-04-16T13:56:34.71+0000 lvl=info msg="Plugin registered" pluginId=doitintl-bigquery-datasource
logger=query_data t=2023-04-16T13:56:34.75+0000 lvl=info msg="Query Service initialization"
logger=live.push_http t=2023-04-16T13:56:34.76+0000 lvl=info msg="Live Push Gateway initialization"
logger=http.server t=2023-04-16T13:56:35+0000 lvl=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=ngalert t=2023-04-16T13:56:35+0000 lvl=info 
msg="warming cache for startup"
logger=ngalert.multiorg.alertmanager t=2023-04-16T13:56:35+0000 lvl=info msg="starting MultiOrg Alertmanager"
logger=grafanaStorageLogger t=2023-04-16T13:56:35+0000 lvl=info msg="storage starting"
logger=server t=2023-04-16T13:56:35+0000 lvl=eror msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=tracing t=2023-04-16T13:56:35+0000 lvl=info msg="Closing tracing"
logger=ngalert t=2023-04-16T13:56:35+0000 lvl=eror msg="unable to fetch orgIds" msg="context canceled"
logger=plugin.grafana-image-renderer t=2023-04-16T13:56:35+0000 lvl=warn msg="plugin failed to exit gracefully"
logger=infra.usagestats.collector t=2023-04-16T13:56:35+0000 lvl=eror msg="Failed to get system stats" error="context canceled"

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.
logger=server t=2023-04-16T13:58:16.08+0000 lvl=eror msg="Server shutdown" error="*rendering.RenderingService run error: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."

Environment: GKE 1.23.16-gke.1400 kube-prometheus-stack-36.2.0 Grafana 9.4.2

What's your helm version?

v3.10.2

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"darwin/amd64"} Kustomize Version: v4.5.7

Which chart?

kube-prometheus-stack

What's the chart version?

kube-prometheus-stack-36.2.0

What happened?

Previously Grafana's versions had a feature that able users to send images/charts embedded in the alert messages. The configuration was something like that:

Recently, the feature was removed and replaced by a custom plugin, to render the images in the same way:

https://grafana.com/grafana/plugins/grafana-image-renderer/

grafana-cli plugins install grafana-image-renderer

Unfortunately, we've had issues with the plugin, it seems some versions (we tested with 9.0.1 - 9.4.2), and we got the errors below:

logger=plugin.manager t=2023-04-16T13:56:34.71+0000 lvl=info msg="Plugin registered" pluginId=doitintl-bigquery-datasource
logger=query_data t=2023-04-16T13:56:34.75+0000 lvl=info msg="Query Service initialization"
logger=live.push_http t=2023-04-16T13:56:34.76+0000 lvl=info msg="Live Push Gateway initialization"
logger=http.server t=2023-04-16T13:56:35+0000 lvl=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=ngalert t=2023-04-16T13:56:35+0000 lvl=info 
msg="warming cache for startup"
logger=ngalert.multiorg.alertmanager t=2023-04-16T13:56:35+0000 lvl=info msg="starting MultiOrg Alertmanager"
logger=grafanaStorageLogger t=2023-04-16T13:56:35+0000 lvl=info msg="storage starting"
logger=server t=2023-04-16T13:56:35+0000 lvl=eror msg="Stopped background service" service=*rendering.RenderingService reason="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."
logger=tracing t=2023-04-16T13:56:35+0000 lvl=info msg="Closing tracing"
logger=ngalert t=2023-04-16T13:56:35+0000 lvl=eror msg="unable to fetch orgIds" msg="context canceled"
logger=plugin.grafana-image-renderer t=2023-04-16T13:56:35+0000 lvl=warn msg="plugin failed to exit gracefully"
logger=infra.usagestats.collector t=2023-04-16T13:56:35+0000 lvl=eror msg="Failed to get system stats" error="context canceled"

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.
logger=server t=2023-04-16T13:58:16.08+0000 lvl=eror msg="Server shutdown" error="*rendering.RenderingService run error: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."

Environment: GKE 1.23.16-gke.1400 kube-prometheus-stack-36.2.0 Grafana 9.4.2

What you expected to happen?

Grafana must be able to send images embedded in the alert message, like that:

How to reproduce it?

Install the helm chart, using the version provided, and try to configure an alert to a Discord channel.

The output will be a text message, without the image/chart:

Enter the changed values of values.yaml?

NONE

Enter the command that you execute and failing/misfunctioning.

grafana-cli plugins install grafana-image-renderer

Anything else we need to know?

No response

[zeritti]

It looks like the image-renderer plugin cannot run in the Grafana image (Ref.):

This plugin is not compatible with the current Grafana Docker image and requires additional system-level dependencies. We recommend setting up another Docker container for rendering and using remote rendering instead. For instruction, refer to Run in Docker.

If you still want to install the plugin with the Grafana Docker image, refer to the instructions on building a custom Grafana image in Grafana Docker documentation.

The Grafana chart itself supports remote rendering (Ref.) through the value imageRenderer (the link points to 6.31 as defined in 36.2.0).

[ccruz2]

Any work arounds for this? I'm currently using kube-prometheus-stack-42.1.1 and having a similar issue, I need to install the plugin for grafana image renderer as well. I tried creating a pod for the image renderer but cant seem to map grafana to use it. Any advice?

[QuentinBisson]

@ccruz2 have you tried to build a custom grafana image as documented in @zeritti 's comment?

[ccruz2]

@QuentinBisson just tried that now, built a custom grafana image with built in image renderer then used that image in my helm chart but image renderer plugin still does not show in grafana.

[zeritti]

@ccruz2 The image build process with provided Dockerfile installs the plugin in directory /var/lib/grafana-plugins through GF_PATHS_PLUGINS:

ENV GF_PATHS_PLUGINS="/var/lib/grafana-plugins"

As a result, the plugin does not show up when using e.g. grafana-cli plugins ls (unless one sets option --pluginsDir) or in the console since it is expected in /var/lib/grafana/plugins as set in the default config file (Ref.). As all the prerequisite software is present in the image, I'd probably list the plugin in grafana.plugins for an install in the plugins directory.

[ccruz2]

Hi @zeritti I can't seem to find /var/lib/grafana/plugins or grafana,plugins at that inside my grafana pod :(

I'm starting to wonder if it's not working because I don't have grafana cli installed on the machine I'm building the image.

Update: I seem to be having an issue using my custom grafana image in kube-prometheus-stack helm chart. Upon checking on a stand alone docker instance the plugin is indeed visible using your suggestion.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.