search

grafana / grafana-image-renderer

A Grafana backend plugin that handles rendering of panels & dashboards to PNGs using headless browser (Chromium/Chrome)
Apache License 2.0
371 stars 151 forks source link

Full page screenshots not working with height != -1 for version >=3.8.4, "Error while waiting for the panels to loads" #488

Open zifter opened 8 months ago

zifter commented 8 months ago

What happened: The latest version of grafana-image-renderer failed to render any dashboards in less than 60 seconds. {"level":"debug","message":"Waiting for dashboard/panel to load","timeout":"60s"} {"level":"debug","message":"Browser request","method":"POST","url":"http://grafana:3000/api/frontend-metrics"} {"level":"debug","message":"Browser request finished","method":"POST","url":"http://grafana:3000/api/frontend-metrics"} {"err":"TimeoutError: Waiting failed: 60000ms exceeded\n at Timeout.<anonymous> (/usr/src/app/node_modules/puppeteer-core/lib/cjs/puppeteer/common/WaitTask.js:59:37)\n at listOnTimeout (node:internal/timers:569:17)\n at process.processTimers (node:internal/timers:512:7)","level":"error","message":"Error while waiting for the panels to load","url":"http://grafana:3000/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=1024&render=1&render=1"} {"level":"debug","message":"Setting zoom level","zoomLevel":1} {"filePath":"/tmp/62cbb3db.png","level":"debug","message":"Taking screenshot"} {"level":"error","message":"Request failed","stack":"Error: Request aborted\n at onaborted (/usr/src/app/node_modules/express/lib/response.js:1052:15)\n at Immediate._onImmediate (/usr/src/app/node_modules/express/lib/response.js:1094:9)\n at process.processImmediate (node:internal/timers:476:21)","url":"/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=1024&renderKey=mCgaaFqDE8KBALUazDrwOcTKnB3man0v&timeout=60&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fd%2FiUfmr5kMk%2Fprometheus-2-22%3ForgId%3D1%26width%3D1024%26height%3D1024%26render%3D1%26render%3D1&width=1024"} {"err":"TimeoutError: Waiting failed: 60000ms exceeded\n at Timeout.<anonymous> (/usr/src/app/node_modules/puppeteer-core/lib/cjs/puppeteer/common/WaitTask.js:59:37)\n at listOnTimeout (node:internal/timers:569:17)\n at process.processTimers (node:internal/timers:512:7)","level":"error","message":"Error while waiting for the panels to load","url":"http://grafana:3000/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=1024&render=1&render=1"} {"level":"debug","message":"Setting zoom level","zoomLevel":1} {"filePath":"/tmp/6ee09d17.png","level":"debug","message":"Taking screenshot"} {"file":"/tmp/6ee09d17.png","level":"debug","message":"Deleting temporary file"} {"level":"debug","message":"::ffff:172.27.0.4 - - [07/Jan/2024:10:02:17 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=1024&renderKey=ORjNjvnO5NWOynWvPJsWwod1WiqRmPFn&timeout=60&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fd%2FiUfmr5kMk%2Fprometheus-2-22%3ForgId%3D1%26width%3D1024%26height%3D1024%26render%3D1%26render%3D1&width=1024 HTTP/1.1\" 200 157291 \"-\" \"Grafana/10.1.4\"\n"} {"level":"debug","message":"Connection closed","url":"http://grafana:3000/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=1024&render=1&render=1"} What you expected to happen: The latest grafana-image-renderer renders any dashboards correctly within a reasonable time range.

How to reproduce it (as minimally and precisely as possible):

  1. Download https://github.com/zifter/unireport/
  2. Enters dev folders (cd dev)
  3. make up
  4. open http://localhost:3000/render/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=1024&render=1

Anything else we need to know?: It's working as expected in 3.8.3, but starts failing after 3.8.4 version. I suppose the reason is changes made in https://github.com/grafana/grafana-image-renderer/pull/472/files. panel-container class can't be found, but I see a dynamic style like css-1ijfwvk-panel-container. I suppose it's not working with the latest Grafana setup.

Environment:

zifter commented 7 months ago

Hi, @AgnesToulet! Sorry for disturbing you! I suppose you might be familiar with root cause of that issue and might help me :)

AgnesToulet commented 7 months ago

Hi @zifter, it seems like you are not using the image renderer as it is intended. I followed the steps you wrote to reproduce the issue and it seems like you want to render a screenshot of the full dashboard. If this is indeed the case, you should use height=-1 so use the following URL: http://localhost:3000/render/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=-1&render=1

In this case, it shouldn't timeout.

zifter commented 7 months ago

@AgnesToulet thanks for the replay! Yes, I want to render the whole dashboard, but in my integration test, I specified a fixed height and it worked before 3.8.3, and after 3.8.4 it's working as I described above. It seems like some kind of regression bug for me.

I tested: http://localhost:3000/render/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=-1&render=1 - working http://localhost:3000/render/d/iUfmr5kMk/prometheus-2-22?orgId=1&width=1024&height=1024&render=1 - query more than 60 seconds

Maybe there is some wrong waiting condition for the dashboard to finish loading? Because it's ok to render image with specified width and height

AgnesToulet commented 7 months ago

I'd say, it's more a behavior that was working before but that we never intended to support as you should specify height=-1 to render a full dashboard screenshot.

You're right to say this is related to this change but it fixes the condition that was broken before and didn't wait for anything. The condition to support waiting for the full dashboard to load is written before but it's used only if height=-1.

It could make sense to add a fullPage parameter to both support setting a height and taking a screenshot of the full dashboard. Let me bring this back to the team and if we decide it would be a good addition to the image renderer, contributions to make this happen would be welcomed!

giuliomagnifico commented 2 months ago

Hi, I'm getting the same error for very long screenshot (5000px&scale=2) but only from last update I think? I don't remember since I have an automation and I noticed the bug only now. It was working one month ago, I have tons of long screenshots.

node[8404]: {"error":"Error: Page crashed!","level":"error","message":"Browser page crashed"}

Full output:

service grafana-image-renderer status
● grafana-image-renderer.service - grafana-image-renderer
     Loaded: loaded (/etc/systemd/system/grafana-image-renderer.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-06-26 15:03:06 CEST; 2min 12s ago
   Main PID: 8404 (node)
      Tasks: 57 (limit: 9272)
        CPU: 1min 19.685s
     CGroup: /system.slice/grafana-image-renderer.service
             ├─8404 node build/app.js server --port=8081
             ├─8458 /usr/lib/chromium-browser/chromium-browser --enable-pinch --allow-pre-commit-input --disable-background-networking --disable-background-timer-throttling --disabl
e-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --disable-default-ap
ps --disable-dev-shm-usage --disable-extensions --disable-features=Translate,BackForwardCache,AcceptCHFrame,MediaRouter,OptimizationHints --disable-hang-monitor --disable-ipc-flooding-protectio
n --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --enable-automation --enable-blink-features=IdleDetection --enable-features=NetworkServiceI
nProcess2 --export-tagged-pdf --force-color-profile=srgb --metrics-recording-only --no-first-run --password-store=basic --use-mock-keychain --headless --hide-scrollbars --mute-audio about:blank
 --no-sandbox --disable-gpu --remote-debugging-port=0 --user-data-dir=/tmp/puppeteer_dev_chrome_profile-KqbYch
             ├─8471 "/usr/lib/chromium-browser/chromium-browser --type=zygote --no-zygote-sandbox --no-sandbox --headless --headless"
             ├─8472 "/usr/lib/chromium-browser/chromium-browser --type=zygote --no-sandbox --headless --headless"
             ├─8488 "/usr/lib/chromium-browser/chromium-browser --type=gpu-process --no-sandbox --disable-dev-shm-usage --disable-breakpad --headless --ozone-platform=headless --use
-angle=swiftshader-webgl --headless --gpu-preferences=WAAAAAAAAAAgAAAMAAAAAAAAAAAAAAAAAABgAAAAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGAAAAAAAAAAYAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA
IAAAAAAAAAA== --use-gl=angle --use-angle=swiftshader-webgl --shared-files --field-trial-handle=3,i,16624079584223773722,9105526953188295762,262144 --enable-features=NetworkServiceInProcess2 --d
isable-features=AcceptCHFrame,BackForwardCache,MediaRouter,OptimizationHints,PaintHolding,Translate --variations-seed-version"
             └─8492 "/usr/lib/chromium-browser/chromium-browser --type=renderer --headless --no-sandbox --disable-dev-shm-usage --disable-background-timer-throttling --disable-break
pad --enable-automation --force-color-profile=srgb --remote-debugging-port=0 --allow-pre-commit-input --ozone-platform=headless --disable-gpu-compositing --enable-blink-features=IdleDetection -
-lang=en-US --num-raster-threads=2 --enable-main-frame-before-activation --renderer-client-id=3 --time-ticks-at-unix-epoch=-1719402992245292 --launch-time-ticks=4034529092 --shared-files=v8_con
text_snapshot_data:100 --field-trial-handle=3,i,16624079584223773722,9105526953188295762,262144 --enable-features=NetworkServiceInProcess2 --disable-featur"

Jun 26 15:03:06 DietPi systemd[1]: Started grafana-image-renderer.service - grafana-image-renderer.
Jun 26 15:03:07 DietPi node[8404]: {"level":"info","message":"HTTP Server started, listening at http://localhost:8081"}
Jun 26 15:04:23 DietPi node[8404]: {"error":"Error: Page crashed!","level":"error","message":"Browser page crashed"}