search

grafana / grafana-image-renderer

A Grafana backend plugin that handles rendering of panels & dashboards to PNGs using headless browser (Chromium/Chrome)
Apache License 2.0
384 stars 155 forks source link

image rendering times out sporadically #582

Open jluebbe opened 6 days ago

jluebbe commented 6 days ago

What happened: After updating grafana (to 11.3.1) and the grafana-image-renderer (to 3.11.6), some image rendering requests fail with a timeout, while most work. As far as I can tell it affects ~25% of requests.

The log message from the renderer is:

logger=plugin.grafana-image-renderer t=2024-11-20T20:22:04.460896286+01:00 level=debug msg="Render request received" url="http://127.0.0.1:3000/d-solo/zglP5XSWk/node?panelId=4&var-node=74acb92689a8&from=now-1d&width=650&height=350&theme=light&render=1"
logger=plugin.grafana-image-renderer t=2024-11-20T20:22:05.585531541+01:00 level=error msg="Browser request failed" url="http://127.0.0.1:3000/api/ds/query?ds_type=influxdb&requestId=SQR100" method=POST failure=net::ERR_ABORTED
logger=plugin.grafana-image-renderer t=2024-11-20T20:22:34.507991251+01:00 level=error msg="Error while waiting for the panels to load" url="http://127.0.0.1:3000/d-solo/zglP5XSWk/node?panelId=4&var-node=74acb92689a8&from=now-1d&width=650&height=350&theme=light&render=1" err="TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed
    at CallbackRegistry.clear (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/common/CallbackRegistry.js:72:36)
    at CdpCDPSession._onClosed (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/CDPSession.js:101:25)
    at Connection.onMessage (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:130:25)
    at WebSocket.<anonymous> (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/node/NodeWebSocketTransport.js:44:32)
    at callListener (/snapshot/src/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onMessage (/snapshot/src/node_modules/ws/lib/event-target.js:209:9)
    at WebSocket.emit (node:events:537:28)
    at Receiver.receiverOnMessage (/snapshot/src/node_modules/ws/lib/websocket.js:1220:20)
    at Receiver.emit (node:events:537:28)
    at Immediate.<anonymous> (/snapshot/src/node_modules/ws/lib/receiver.js:601:16)"
logger=plugin.grafana-image-renderer t=2024-11-20T20:22:34.508769665+01:00 level=error msg="Render request failed" url="http://127.0.0.1:3000/d-solo/zglP5XSWk/node?panelId=4&var-node=74acb92689a8&from=now-1d&width=650&height=350&theme=light&render=1" error="Error: Timeout hit: 30000"

The influx log doesn't show anything obviously problematic.

What you expected to happen: All requests render correctly.

How to reproduce it (as minimally and precisely as possible): The images are used on public pages: https://freifunk-bs.de/map/#!/en/map/68725132e5cb

Anything else we need to know?: In the error case, the renderer responds with a truncated error message instead of the image:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta name="viewport" content="width=device-width" />
    <meta name="theme-color" content="#000" />

    <title>Grafana - Error</title>

    <base href="/grafana/" />

    <link rel="stylesheet" href="

Environment:

mikerevellesciplay commented 1 day ago

had the same issue with 11.3.1, spent more than a week troubleshooting this and it looks like downgrading to 11.2.4 fixed it. directly rendering from the grafana interface (panel -> share -> generate rendered image) works fine but rendering on alert is broken.