Plugin not working with Grafana API

Adrian-LSY commented 3 months ago

Hello, I've been encountering issues generating PDFs using service accounts and the Grafana API. In my Grafana app, I have several orgs. Generating PDFs from the browser is working as intended tho.

I'm currently on Grafana version 10.3.4 Plugin version 1.5.0

At the moment I am trying to generate a pdf from the dashboard c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 using the service account glsa_<token> This dashboard is found in the org id 4

Here are some details about the service account.

curl -H "Authorization: Bearer <token>" -X GET 'http://localhost:3000/api/access-control/user/permissions'

{
  "annotations:create": [
    "annotations:type:dashboard"
  ],
  "annotations:delete": [
    "annotations:type:dashboard"
  ],
  "annotations:read": [
    "annotations:*"
  ],
  "annotations:write": [
    "annotations:type:dashboard"
  ],
  "dashboards:read": [
    "dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6"
  ],
  "datasources.id:read": [
    "datasources:*"
  ],
  "datasources:query": [
    "datasources:*",
    "datasources:uid:grafana"
  ],
  "datasources:read": [
    "datasources:*",
    "datasources:uid:grafana"
  ],
  "library.panels:read": [
    "folders:uid:general"
  ],
  "orgs.quotas:read": [
    ""
  ],
  "orgs:read": [
    ""
  ],
  "plugins.app:access": [
    "plugins:*"
  ]
}

Requesting the endpoint

$ curl -o report.pdf -H "Authorization: Bearer <token>" "http://localhost:3000/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report?dashUid=dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100    24    0    24    0     0      5      0 --:--:--  0:00:04 --:--:--     6

$ cat report.pdf

error generating report

The API returns: error generating report

Here's logs of when the the request was sent

It seems that the plugin is trying to grab the dashboard from the org id of 1 which returns a 404 and tries to authenticate the account with an org id of 0


grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:20.446670256Z level=debug msg="no variables found" dash_uid=dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-reporting

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:20.446724136Z level=debug msg="time range" endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app range="map[From:now-1h To:now]" user=sa-reporting dash_uid=dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:20.446887003Z level=info msg="updated config" config="Grafana App URL: http://localhost:3000; Skip TLS Check: false; Grafana Data Path: /var/lib/grafana; Orientation: landscape; Layout: grid; Dashboard Mode: full; Time Zone: ; Encoded Logo: ; Max Renderer Workers: 2; Persist Data: false; Included Panel IDs: all; Excluded Panel IDs: none" dash_uid=dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-reporting

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:20.525016712Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.0.237 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:20.525160181Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.0.237 time_ms=66 duration=66.99388ms size=46 referer= handler=/api/dashboards/uid/:uid

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T03:58:22.685304399Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.0.237 traceID=

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T03:58:22.686641284Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.0.237 time_ms=1 duration=1.229986ms size=102 referer= handler=/api/live/ws

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.318566947Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.0.237 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.318658928Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.0.237 time_ms=85 duration=85.686314ms size=46 referer=http://localhost:3000/d/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.41560385Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.0.237 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.415706886Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.0.237 time_ms=94 duration=94.248505ms size=46 referer=http://localhost:3000/d/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T03:58:23.718358061Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.0.237 traceID=

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T03:58:23.718970707Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.0.237 time_ms=0 duration=642.174µs size=102 referer= handler=/api/live/ws

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.722980148Z level=info msg="Request Completed" method=GET path=/api/user/orgs status=403 remote_addr=10.0.0.237 time_ms=194 duration=194.884201ms size=47 referer="http://localhost:3000/d/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/user/orgs

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.823860452Z level=error msg="Invalid dashboard UID in annotation request" error="Dashboard not found" remote_addr=10.0.0.237 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:23.824173392Z level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=10.0.0.237 time_ms=62 duration=62.52623ms size=70 referer="http://localhost:3000/d/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/annotations

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T03:58:24.455968332Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6: unexpected end of JSON input\nerror obtaining dashboard from http://localhost:3000/api/dashboards/uid/dashboards:uid:c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6. Got Status 404 Not Found, message: {\"message\":\"Dashboard not found\",\"traceID\":\"\"} " pluginID=mahendrapaipuri-dashboardreporter-app

grafana-instance | logger=context userId=45 orgId=4 uname=sa-reporting t=2024-08-12T03:58:24.45658772Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.1.166 time_ms=4070 duration=4.070148999s size=24 referer= handler=/api/plugins/:pluginId/resources/*

mahendrapaipuri commented 3 months ago

Hello,

You will have to pass orgId=4 query parameter as well. I see that you have dashboards:uid prefix before the actual UID of the dashboard. It should not be there. In your case, the request should be $ curl -o report.pdf -H "Authorization: Bearer <token>" "http://localhost:3000/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report?dashUid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6&orgId=4".

PS: Please redact all sensitive info before posting on public forums like GitHub issues.

Adrian-LSY commented 3 months ago

thanks for the reply and I appreciate that you went the extra mile to redact sensitive information, all information in the initial post (auth token) was dummy data and does not represent any actual information

I've tried your suggestion mentioned above and had a look at a previous closed issue that's similar to mine

Here's the new curl request

$ curl -o report.pdf -H "Authorization: Bearer <some_token>" "http://localhost:3000/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report?dashUid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6&orgId=4"

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100    24    0    24    0     0      5      0 --:--:--  0:00:04 --:--:--     5

$ cat report.pdf

error generating report

Here are the logs

They look generally the same as before. The passed orgId=4 seems to have no effect

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:04.64294194Z level=debug msg="no variables found" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-kgpa-reporting

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:04.643060767Z level=debug msg="time range" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app range="map[From:now-1h To:now]" user=sa-kgpa-reporting

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:04.643205634Z level=info msg="updated config" pluginID=mahendrapaipuri-dashboardreporter-app user=sa-kgpa-reporting config="Grafana App URL: http://localhost:3000; Skip TLS Check: false; Grafana Data Path: /var/lib/grafana; Orientation: landscape; Layout: grid; Dashboard Mode: full; Time Zone: ; Encoded Logo: ; Max Renderer Workers: 2; Persist Data: false; Included Panel IDs: all; Excluded Panel IDs: none" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:04.757039534Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:04.757098961Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=85 duration=85.925843ms size=46 referer= handler=/api/dashboards/uid/:uid

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T07:26:07.356631802Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T07:26:07.356788683Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=0 duration=234.679µs size=102 referer= handler=/api/live/ws

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.107763372Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.107891262Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=182 duration=182.681437ms size=46 referer=http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.202465565Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.202534133Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=85 duration=85.874611ms size=46 referer=http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T07:26:08.411148549Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T07:26:08.411217562Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=0 duration=117.928µs size=102 referer= handler=/api/live/ws

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.541080278Z level=error msg="Invalid dashboard UID in annotation request" error="Dashboard not found" remote_addr=10.0.1.166 traceID=

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.541223643Z level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=10.0.1.166 time_ms=95 duration=95.493535ms size=70 referer="http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/annotations

grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:08.565414839Z level=info msg="Request Completed" method=GET path=/api/user/orgs status=403 remote_addr=10.0.1.166 time_ms=275 duration=275.159788ms size=47 referer="http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/user/orgs

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T07:26:09.239648485Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6: unexpected end of JSON input\nerror obtaining dashboard from http://localhost:3000/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6. Got Status 404 Not Found, message: {\"message\":\"Dashboard not found\",\"traceID\":\"\"} " pluginID=mahendrapaipuri-dashboardreporter-app

grafana-instance | logger=context userId=46 orgId=4 uname=sa-kgpa-reporting t=2024-08-12T07:26:09.242507183Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.1.166 time_ms=4680 duration=4.680180103s size=24 referer= handler=/api/plugins/:pluginId/resources/*

grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T07:26:09.914040036Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=7 duration=7.338574ms size=105 referer= handler=/api/live/ws

mahendrapaipuri commented 3 months ago

Did you enable the plugin for Org 4? The provisioning should be made for each organization separately. I sense that the plugin is using the service token provisioned for Org 1 to make API requests to the resources on Org 4.

Adrian-LSY commented 3 months ago

I believe I enabled the plugin app-wide. How would I make separate provisioning for each organization?

mahendrapaipuri commented 3 months ago

Plugin apps are enabled only Org wide. A sample config file can be:

apiVersion: 1

apps:
  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 1
    org_name: <Org name>
    disabled: false
    secureJsonData:
       <snippedt>
    jsonData:
       <snippet>

- type: mahendrapaipuri-dashboardreporter-app
   org_id: 4
   org_name: <Org name>
   disabled: false
   secureJsonData:
       <snippedt>
   jsonData:
       <snippet>

Adrian-LSY commented 3 months ago

Alright, here is my version of the config. Following the instructions in the readme and the sample config in the repo, here's what I have at the moment

apiVersion: 1

apps:
  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 1
    org_name: Main Org.
    disabled: false

  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 4
    org_name: foobar
    disabled: false
    secureJsonData:
      saToken: 'some-token'

I've added it to my docker container /etc/grafana/provisioning/plugins/reporter.yaml I've restarted the container and ran the curl command again with the same results.

I'm quite certain I'm missing something here.

mahendrapaipuri commented 3 months ago

How did you get some-token that you are using in the config?

If you are using externalServiceAccounts feature flag, there is no need to pass any token to the plugin. And you need to create token for each org separately. You cannot use token created in Org 1 to make requests to Org 4. Grafana identifies the Org based on the API token you use.

Here should be your workflow:

Enable externalServiceAccounts feature flag in your Grafana instance
Provision the plugin for both orgs with the config you posted (you can remove secureJsonData block tho)
Switch Org to Org 4 and create a new service account and a token
Use this token to make API requests to plugin

Adrian-LSY commented 3 months ago

Alright i'll try to document the process as best as I can

Here's the contents of my docker-compose

version: '3.0'

services:
  grafana:
    image: custom-grafana-img
    container_name: 'grafana-instance'
    build:
      context: ./.config
      args:
        grafana_image: ${GRAFANA_IMAGE:-grafana-oss}
        grafana_version: ${GRAFANA_VERSION:-10.3.4}
    ports:
      - 3000:${GF_SERVER_HTTP_PORT:-3000}/tcp
    environment:
      - GF_FEATURE_TOGGLES_ENABLE=${GF_FEATURE_TOGGLES_ENABLE:-externalServiceAccounts}
      - GF_ALERTING_ENABLED=false
      - GF_UNIFIED_ALERTING_ENABLED=false
      - GF_RENDERING_SERVER_URL=http://renderer:8081/render
      - GF_RENDERING_CALLBACK_URL=https://grafana:${GF_SERVER_HTTP_PORT:-3000}/
      - "GF_LOG_FILTERS=rendering:debug plugin.mahendrapaipuri-dashboardreporter-app:debug"
  renderer:
    image: grafana/grafana-image-renderer:latest
    environment:
      - RENDERING_MODE=clustered
      - RENDERING_CLUSTERING_MODE=browser
      - RENDERING_CLUSTERING_MAX_CONCURRENCY=5
      - RENDERING_CLUSTERING_TIMEOUT=60
      - IGNORE_HTTPS_ERRORS=true
    ports:
      - 8081
  chrome:
    image: chromedp/headless-shell:latest
    shm_size: 2G
    init: true
    network_mode: service:grafana

reporter.yaml

apiVersion: 1

apps:
  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 1
    org_name: Main Org.
    disabled: false

  - type: mahendrapaipuri-dashboardreporter-app
    org_id: 4
    org_name: foobar
    disabled: false

Here's the workflow I followed: ran docker-compose down and up copied the contents of reporter.yaml to /etc/grafana/provisioning/plugins/reporter.yaml

restarted the container to load the provisioning

created a new service account:

Got the new token and ran the curl command

$   $ curl -o report.pdf -H "Authorization: Bearer <new_token>" "http://localhost:3000/report?dashUid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6&orgId=4"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24    0    24    0     0      6      0 --:--:--  0:00:03 --:--:--     6

$ cat report.pdf

error generating report

Here is the log file I just noticed, the context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app is used to look for the dashboard

The response context is userId=50 orgId=4 uname=sa-reporter which I believe is the newly created service account

grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:53.157682632Z level=debug msg="no variables found" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-reporter
grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:53.159379911Z level=debug msg="time range" range="map[From:now-1h To:now]" user=sa-reporter dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app
grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:53.160408028Z level=info msg="updated config" config="Grafana App URL: http://localhost:3000; Skip TLS Check: false; Grafana Data Path: /var/lib/grafana; Orientation: landscape; Layout: grid; Dashboard Mode: full; Time Zone: ; Encoded Logo: ; Max Renderer Workers: 2; Persist Data: false; Included Panel IDs: all; Excluded Panel IDs: none" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-reporter
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:53.289254366Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:53.289351636Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=106 duration=106.736175ms size=46 referer= handler=/api/dashboards/uid/:uid
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:55.369794652Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=10.0.0.237 time_ms=0 duration=114.763µs size=29 referer= handler=/
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:55.770446227Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:55.784361414Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=14 duration=14.068967ms size=102 referer= handler=/api/live/ws
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:56.495790043Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:56.498351104Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=128 duration=128.047016ms size=46 referer=http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:56.639300346Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:56.639655636Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=137 duration=137.018825ms size=46 referer=http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:56.862237719Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:56.86644831Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=4 duration=4.251835ms size=102 referer= handler=/api/live/ws
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:57.004563578Z level=info msg="Request Completed" method=GET path=/api/user/orgs status=403 remote_addr=10.0.1.166 time_ms=301 duration=301.267318ms size=47 referer="http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/user/orgs
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:57.054825056Z level=error msg="Invalid dashboard UID in annotation request" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:57.054906647Z level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=10.0.1.166 time_ms=147 duration=147.820041ms size=70 referer="http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/annotations
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:57.517386031Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=
grafana-instance | logger=context userId=0 orgId=0 uname= t=2024-08-12T10:06:57.517629356Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=0 duration=279.409µs size=102 referer= handler=/api/live/ws
grafana-instance | logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-12T10:06:57.738474604Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6: unexpected end of JSON input\nerror obtaining dashboard from http://localhost:3000/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6. Got Status 404 Not Found, message: {\"message\":\"Dashboard not found\",\"traceID\":\"\"} " pluginID=mahendrapaipuri-dashboardreporter-app
grafana-instance | logger=context userId=50 orgId=4 uname=sa-reporter t=2024-08-12T10:06:57.7397879Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.237 time_ms=4697 duration=4.697234391s size=24 referer= handler=/api/plugins/:pluginId/resources/*

mahendrapaipuri commented 3 months ago

I can reproduce the beahviour you are reporting and this is a limitation of development environment. I ll try to explain.

When we enable externalServiceAccounts feature, Grafana will create a service account and provision a token for the plugin at the start up. When we interact with the plugin via API, this provisioned token is used to make API requests to Grafana. So, when we manually create a second organization from Grafana UI and enable the plugin, there is no service account created for the plugin in this organization. Because it seems like service accounts are only created at Grafana's startup. I dont think there is a way to restart Grafana server inside the container as it has PID 1. And AFAIK there is no way to provision Orgs in Grafana yet.

So, the only way to test this is to persist the changes you make to Grafana (by storing DB on a persistent storage) and restart the Grafana instance after creating new Org. Then you will notice Grafana creating a service account for the plugin in a new Org.

Adrian-LSY commented 3 months ago

I'm already using Postgres to store Grafana's data. I created a new organization, enabled the plugin for the organization from the web UI and restarted the docker container. I created a new dashboard and service account and ran the curl command again. This still did not resolve the issue

mahendrapaipuri commented 3 months ago

Ok, one more thing to verify. After you have created a new org, enabled plugin and restarted Grafana instance do you see a service account created by Grafana for the plugin? You will see something like this:

This is the external service account created by Grafana for the plugin to use. You will/should have a similar account in the default Org. too. Could you check this please?

Adrian-LSY commented 3 months ago

Following the process...

I created a new test org

The plugin is disabled by default

Enabling it...

And then restarting the containers

 *  Executing task: docker-compose --file 'docker-compose.yml' --project-name 'admin' restart 

Restarting grafana-instance ... done
Restarting renderer_1  ... done
 *  Terminal will be reused by tasks, press any key to close it.

No new service account was created for the new organization

mahendrapaipuri commented 3 months ago

Just one last check: After you restart the Grafana instance, what is the state of the plugin? Still enabled?

Adrian-LSY commented 3 months ago

Yep just checked.

mahendrapaipuri commented 3 months ago

Ahhh, I am afraid that this is a bug in Grafana then! As a workaround, you can manually configure a service token in the Service Account Token field that you see in the plugin's config.

Adrian-LSY commented 3 months ago

Alright, on org 4, i've generated a new token and added that to the service access token in the plugin's settings. The result is more or less the same.

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:00.253790485Z level=debug msg="no variables found" endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-reporter dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:00.253846004Z level=debug msg="time range" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app range="map[From:now-1h To:now]" user=sa-reporter
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:00.254657096Z level=info msg="updated config" config="Grafana App URL: http://localhost:3000; Skip TLS Check: false; Grafana Data Path: /var/lib/grafana; Orientation: landscape; Layout: grid; Dashboard Mode: full; Time Zone: ; Encoded Logo: ; Max Renderer Workers: 2; Persist Data: false; Included Panel IDs: all; Excluded Panel IDs: none" dash_uid=c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=sa-reporter
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:00.357842785Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:00.358190899Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=98 duration=98.537814ms size=46 referer= handler=/api/dashboards/uid/:uid
logger=context userId=0 orgId=0 uname= t=2024-08-13T09:33:03.360225574Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=
logger=context userId=0 orgId=0 uname= t=2024-08-13T09:33:03.360875669Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=0 duration=699.521µs size=102 referer= handler=/api/live/ws
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:03.943440383Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:03.943526149Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=121 duration=121.199342ms size=46 referer=http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:04.1379768Z level=error msg="Dashboard not found" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:04.138181546Z level=info msg="Request Completed" method=GET path=/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6 status=404 remote_addr=10.0.1.166 time_ms=190 duration=190.58941ms size=46 referer=http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_? handler=/api/dashboards/uid/:uid
logger=context userId=0 orgId=0 uname= t=2024-08-13T09:33:04.312090076Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=10.0.1.166 traceID=
logger=context userId=0 orgId=0 uname= t=2024-08-13T09:33:04.312454706Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.1.166 time_ms=0 duration=207.641µs size=102 referer= handler=/api/live/ws
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:04.378899909Z level=info msg="Request Completed" method=GET path=/api/user/orgs status=403 remote_addr=10.0.1.166 time_ms=195 duration=195.687786ms size=47 referer="http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/user/orgs
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:04.457238294Z level=error msg="Invalid dashboard UID in annotation request" error="Dashboard not found" remote_addr=10.0.1.166 traceID=
logger=context userId=43 orgId=1 uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:04.457309757Z level=info msg="Request Completed" method=GET path=/api/annotations status=400 remote_addr=10.0.1.166 time_ms=95 duration=95.787316ms size=70 referer="http://localhost:3000/d/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6/_?orgId=1" handler=/api/annotations
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-08-13T09:33:05.14977758Z level=error msg="error generating report" endpoint=callResource err="error fetching dashboard c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6: unexpected end of JSON input\nerror obtaining dashboard from http://localhost:3000/api/dashboards/uid/c1bd0ad3-cd1b-46a5-b29f-6d33233c6ff6. Got Status 404 Not Found, message: {\"message\":\"Dashboard not found\",\"traceID\":\"\"} " pluginID=mahendrapaipuri-dashboardreporter-app
logger=context userId=50 orgId=4 uname=sa-reporter t=2024-08-13T09:33:05.152010691Z level=error msg="Request Completed" method=GET path=/api/plugins/mahendrapaipuri-dashboardreporter-app/resources/report status=500 remote_addr=10.0.0.237 time_ms=4972 duration=4.972380886s size=24 referer= handler=/api/plugins/:pluginId/resources/*

mahendrapaipuri commented 3 months ago

I can confirm this too. Again, this is a bug on Grafana side.

Seems like Grafana is sending the token created in Org 1 when we make request to Org 4. That is why you see the username identified in the requests logs uname=sa-1-extsvc-mahendrapaipuri-dashboardreporter-app correspond to the service account in Org 1. The plugin always prioritizes the service account token to the one configured manually. I have filed an issue on Grafana, let's see if they confirm this.

As a workaround, you can disable externalServiceAccounts feature and manually configure the token for each Org. I have tested this approach both using browser and making a API request and it worked.

Adrian-LSY commented 3 months ago

Removing externalServiceAccounts and manually configuring the token seems to do the trick.

mahendrapaipuri / grafana-dashboard-reporter-app