search

grafana / grafana-image-renderer

A Grafana backend plugin that handles rendering of panels & dashboards to PNGs using headless browser (Chromium/Chrome)
Apache License 2.0
382 stars 154 forks source link

401 (Unauthorized) remote render service when running as remote docker container #267

Open Bashere1 opened 3 years ago

Bashere1 commented 3 years ago

What happened: I am running grafana via container utilizing google compute engine. https://cloud.google.com/compute/docs/containers/deploying-containers

In another separate google compute engine I am running the remote render image via container using the same model. However, when I try to generate a "Direct link rendered image" render fails.

I noticed in the renderer logs that we are getting a 401 unauthorized to the callback url. {"level":"info","message":"HTTP Server started, listening at http://redacted_ip:8081"} {"msg":"Failed to load resource: the server responded with a status of 401 (Unauthorized)","url":"http://grafana.mydomain.com:3000/d-solo/gL-UXHm7z/mydashboard?orgId=1&from=1628254952332&to=1628276552332&panelId=2&width=1000&height=500&tz=America%2FChicago&render=1","level":"error","message":"Browser console error"} {"url":"/render?deviceScaleFactor=1.000000&domain=grafana.mydomain.com&encoding=&height=500&renderKey=REDACTED&timeout=60&timezone=America%2FChicago&url=http%3A%2F%2Fgrafana.mydomain.com%3A3000%2Fd-solo%2FgL-UXHm7z%2Fmydashboard%3ForgId%3D1%26from%3D1628254952332%26to%3D1628276552332%26panelId%3D2%26width%3D1000%26height%3D500%26tz%3DAmerica%252FChicago%26render%3D1&width=1000","stack":"TimeoutError: waiting for function failed: timeout 60000ms exceeded\n at new WaitTask (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:388:34)\n at DOMWorld.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/DOMWorld.js:303:16)\n at Frame.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/FrameManager.js:402:32)\n at Page.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/Page.js:805:33)\n at Browser. (/usr/src/app/build/browser/browser.js:195:24)\n at Generator.next ()\n at fulfilled (/usr/src/app/build/browser/browser.js:5:58)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)","level":"error","message":"Request failed"}

As expected if I test the URL from my browser while logged in the above url loads successfully. I should also clarify that I have setup LDAP auth for the grafana instance. I've also attempted to set GF_AUTH_ANONYMOUS_ENABLED=true as well with no luck.

What you expected to happen:

Grafana to render images using remote render service

How to reproduce it (as minimally and precisely as possible):

  1. Deploy grafana via docker instance
  2. Deploy grafana image render via docker instance on alternate compute
  3. Attempt to share direct link with rendered images

Anything else we need to know?:

Environment:

EnvironmentVariables GF_RENDERING_SERVER_URL: http://REDACTED:8081/render GF_RENDERING_CALLBACK_URL: http://grafana.mydomain.com:3000 GF_AUTH_ANONYMOUS_ENABLED: true GF_LOG_FILTERS: rendering:debug

AgnesToulet commented 3 years ago

Hello! Thanks for reaching us, do you also have the logs from the Grafana server?

stdevPavelmc commented 2 years ago

I'm hitting a similar Issue today

K8s env but docker images under the hood.

A grafana server & grafana-render with the vars setup like the one on your the docker docs.

401 Unauthorized

Partial grafana debug logs follows

t=2022-09-09T17:42:45+0000 lvl=info msg="New state change" logger=alerting.resultHandler ruleId=8 newState=ok prev state=no_data
t=2022-09-09T17:42:45+0000 lvl=info msg=Rendering logger=rendering renderer=http path="d-solo/h5sY0munk/server-monitoring-general?orgId=1&panelId=22"
t=2022-09-09T17:42:45+0000 lvl=dbug msg="calling remote rendering service" logger=rendering renderer=http url="http://grafana-renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=[.....]&timeout=15&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fgrafana%2Fd-solo%2Fh5sY0munk%2Fserver-monitoring-general%3ForgId%3D1%26panelId%3D22%26render%3D1&width=1000"
t=2022-09-09T17:42:45+0000 lvl=eror msg="Remote rendering request failed" logger=rendering renderer=http error="401 Unauthorized"
t=2022-09-09T17:42:45+0000 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=8 error="Remote rendering request failed. 401: 401 Unauthorized"
t=2022-09-09T17:42:45+0000 lvl=eror msg="Could not generate Telegram message with inline image." logger=alerting.notifier.telegram err="open : no such file or directory"

Grafana render log entry:

{"level":"error","message":"::ffff:10.1.15.92 - - [09/Sep/2022:18:14:15 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=[.....]&timeout=15&timezone=&url=http%3A%2F%2Fgrafana%3A3000%2Fgrafana%2Fd-solo%2Fh5sY0munk%2Fserver-monitoring-general%3ForgId%3D1%26panelId%3D22%26render%3D1&width=1000 HTTP/1.1\" 401 74 \"-\" \"Grafana/7.3.3\"\n"}
ycyin commented 2 years ago

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

romcheck commented 2 years ago

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

3.5.0 version works for me (grafana version is 9.0.5)

kisshore commented 2 years ago

Even for me 3.5.0 worked with Grafana 8.3.3 version

Codecaver commented 2 years ago

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

works for me !! grafana version 8.4.2

yetkinergun commented 2 years ago

I had the same issue - I was already on the latest version of image renderer, solution for me was to upgrade Grafana from v9.1.0 to v9.1.6

carlosrodfern commented 2 years ago

Same here with Grafana 7.5.5 and image renderer "lastest" or "3.6.1". Downgrading image renderer to 3.5.0 worked for me.

debu99 commented 2 years ago

lastest is not working....

KevinGage commented 2 years ago

Same. I was using "latest" tag for grafana and image renderer. I started getting the 401 errors. I followed the instructions here to make sure a token was properly setup. But I am still getting 401 messages in the image renderer docker logs and images wont render. I rolled back to 3.5 and it's working again but now I suspect I am vulnerable.

tomelliff commented 1 year ago

It's worth pointing out that 3.6.0 contains this change which has a requirement on Grafana >=8.3.11 so if you have an older version of Grafana you'll see 401 failures on 3.6.0+/latest.

dmitryp-orca commented 1 year ago

Grafana 8.5.1 + Image rendered 3.6.2 - does not work (Error: Unauthorized request). Probably, grafana-image-renderer v. 3.6.2 (which is the latest at the moment) is broken OR incompatible with Grafana 8 and below. https://hub.docker.com/r/grafana/grafana-image-renderer/tags

If these versions are supposed to be incompatible, this requires a clear error message on that.

tyrken commented 1 year ago

@dmitryp-orca see https://grafana.com/blog/2022/08/30/security-release-new-versions-of-grafana-and-grafana-image-renderer-with-a-high-severity-security-fix-for-cve-2022-31176/ - in the 8.5 series would need at least 8.5.11

sv-hmelevsky commented 1 year ago

Hi i have same issue here.

I have Grafana: 9.4.3 and Grafana Image Renderer: 3.6.4. I use Image Renderer like standalone app: node build/app.js server --port=8081 In my grafana.ini:

[unified_alerting.screenshots]
capture = true

[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
rendering_args = --no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758,--ignore-certificate-errors,--ignore-certificate-errors-spki-list
auth_token = token_string (same as renderer_token)

[rendering]
server_url = http://192.168.171.53:8081/render
callback_url = http://192.168.171.53:3000
server_domain = 192.168.171.53
renderer_token = token_string (same as auth_token)

After start the Image renderer i was got this: 

{"level":"error","message":"::ffff:192.168.171.53 - - [28/Mar/2023:19:56:01 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=xXCCVR82sEGgIhVGtzro8EbtcPdwhX2s&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D20%26to%3Dnow%26render%3D1&width=1000 HTTP/1.1\" 401 74 \"-\" \"Grafana/9.4.3\""}
{"level":"error","message":"Request failed","stack":"Error: Unauthorized request
   at /home/atb/grafana-image-renderer/build/service/middlewares.js:27:30
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9
   at Function.process_params (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:346:12 undefined)
   at next .next (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:280:10 undefined) 
   at jsonParser .jsonParser (/home/atb/grafana-image-renderer/node_modules/body-parser/lib/types/json.js:110:7 undefined)
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   
   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9","url":"/render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=ilFSQne8WdyAC4KuzbwXLz5jhh6r2PwJ&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D68%26to%3Dnow%26render%3D1&width=1000"}

If I add console.log(config) in the middleware.js, i see in console - authToken: '-' (Image renderere in standalone mode dosn't get Grafana config ENV and Settings?) Okay, what if you comment out the check of the token at all. I was get: 

{"err":"TimeoutError: Navigation timeout of 10000 ms exceeded
    at LifecycleWatcher._LifecycleWatcher_createTimeoutPromise (/home/atb/grafana-image-renderer/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:167:12)","level":"error","message":"Error while trying to prepare page for screenshot","url":"http://192.168.171.53:3000/d-solo/A-sK4r-Mk/20_test_ec?from=now-1h&orgId=1&panelId=80&to=now&render=1"}
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit

btw same thing if i remove auth_token from grafana.ini and add - to renderer_token

At the same time, URLs that cannot be reached by GIR open in the browser without problems. What am I doing wrong? How do I get GIR to work with Grafana and get it to see settings.

pelandrun commented 1 year ago

Hi i have same issue here.

I have Grafana: 9.4.3 and Grafana Image Renderer: 3.6.4. I use Image Renderer like standalone app: node build/app.js server --port=8081 In my grafana.ini:

[unified_alerting.screenshots]
capture = true

[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
rendering_args = --no-sandbox,--disable-setuid-sandbox,--disable-dev-shm-usage,--disable-accelerated-2d-canvas,--disable-gpu,--window-size=1280x758,--ignore-certificate-errors,--ignore-certificate-errors-spki-list
auth_token = token_string (same as renderer_token)

[rendering]
server_url = http://192.168.171.53:8081/render
callback_url = http://192.168.171.53:3000
server_domain = 192.168.171.53
renderer_token = token_string (same as auth_token)

After start the Image renderer i was got this:

{"level":"error","message":"::ffff:192.168.171.53 - - [28/Mar/2023:19:56:01 +0000] \"GET /render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=xXCCVR82sEGgIhVGtzro8EbtcPdwhX2s&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D20%26to%3Dnow%26render%3D1&width=1000 HTTP/1.1\" 401 74 \"-\" \"Grafana/9.4.3\""}
{"level":"error","message":"Request failed","stack":"Error: Unauthorized request
   at /home/atb/grafana-image-renderer/build/service/middlewares.js:27:30
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9
   at Function.process_params (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:346:12 undefined)
   at next .next (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:280:10 undefined) 
   at jsonParser .jsonParser (/home/atb/grafana-image-renderer/node_modules/body-parser/lib/types/json.js:110:7 undefined)
   at Layer.handle [as handle_request] (/home/atb/grafana-image-renderer/node_modules/express/lib/router/layer.js:95:5 undefined)  
   at trim_prefix .trim_prefix (/home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:328:13 undefined)   
   at /home/atb/grafana-image-renderer/node_modules/express/lib/router/index.js:286:9","url":"/render?deviceScaleFactor=1.000000&domain=192.168.171.53&encoding=&height=500&renderKey=ilFSQne8WdyAC4KuzbwXLz5jhh6r2PwJ&timeout=10&timezone=&url=http%3A%2F%2F192.168.171.53%3A3000%2Fd-solo%2FZjraN-fGz%2F21_test_ph%3Ffrom%3Dnow-1h%26orgId%3D1%26panelId%3D68%26to%3Dnow%26render%3D1&width=1000"}

If I add console.log(config) in the middleware.js, i see in console - authToken: '-' (Image renderere in standalone mode dosn't get Grafana config ENV and Settings?) Okay, what if you comment out the check of the token at all. I was get:

{"err":"TimeoutError: Navigation timeout of 10000 ms exceeded
    at LifecycleWatcher._LifecycleWatcher_createTimeoutPromise (/home/atb/grafana-image-renderer/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:167:12)","level":"error","message":"Error while trying to prepare page for screenshot","url":"http://192.168.171.53:3000/d-solo/A-sK4r-Mk/20_test_ec?from=now-1h&orgId=1&panelId=80&to=now&render=1"}
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(node:48534) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit

btw same thing if i remove auth_token from grafana.ini and add - to renderer_token

At the same time, URLs that cannot be reached by GIR open in the browser without problems. What am I doing wrong? How do I get GIR to work with Grafana and get it to see settings.

That work form me. I think you have come kind of connection problem between yout render and the grafana server

ccruz2 commented 1 year ago

oding=&height=500&

Was this resolved? getting the same issue

kwenzh commented 1 year ago

the same issue， grafana 9.1.2 ， image renderer 3.7.0

AgnesToulet commented 1 year ago

Hello! @ccruz2 @kwenzh Can you share your Grafana and image renderer setup please?

It seems like a lot of issues in this thread comes from the security update that happened in August 2022. Please ensure you have correctly set up your shared token between Grafana and the image renderer (as explained here: https://grafana.com/blog/2022/08/30/security-release-new-versions-of-grafana-and-grafana-image-renderer-with-a-high-severity-security-fix-for-cve-2022-31176/#:~:text=by%20this%20vulnerability.-,Solutions%20and%20mitigations,-All%20Grafana%20installations). Please note that if you are running the image renderer as a plugin (installed in your Grafana folder), you need to add an auth_token config key in the [plugin.grafana-image-renderer] section of the Grafana config file. But if you run the image renderer as a separate server, you need to set it as an environment variable (AUTH_TOKEN or in the image renderer configuration file (https://github.com/grafana/grafana-image-renderer/blob/master/default.json#L21). In both cases, you will need to also set it In the rendering section of the Grafana configuration file, in renderer_token.

minibear2333 commented 7 months ago

I had the same problem. Try changing the version of Grafana Image Renderer and try again.

it's work! grafana 8.4.2 the image render use 3.4.2

yioda commented 5 months ago

I've tested grafana-image-renderer v3.10.4 with

[Updated 2024/05/16] When I try to upgrade from

emmanuelbertho commented 4 months ago

Same problem I only managed to make it work with Grafana v9.5.1 and image-renderer v3.10.4 as mentionned in @yioda message

teckglobal commented 2 months ago

Same Here with Grafana & Renderer newest docker containers, trying to render the images gives the 401 error

roock commented 4 weeks ago

Just a side node for anyone using the grafana image render in a container and setting the auth token in the config file, the documentation is incomplete/misleading, I've opened a PR to improve the example configuration.