search

mahendrapaipuri / grafana-dashboard-reporter-app

A Grafana plugin app to create PDF reports of dashboards
Apache License 2.0
51 stars 6 forks source link

Update plugin backend to work with Grafana v11.3.0 #154

Closed mahendrapaipuri closed 3 weeks ago

mahendrapaipuri commented 3 weeks ago

Closes #146

mahendrapaipuri commented 3 weeks ago

@wulfchr @J-Paul0815 Could you please test this patch? You can download plugin artifacts from CI here.

If the artifacts are expired, please let me know, I will rerun the workflow.

J-Paul0815 commented 3 weeks ago

@wulfchr @J-Paul0815 Could you please test this patch? You can download plugin artifacts from CI here.

Change Folder mahendrapaipuri-dashboardreporter-app to mahendrapaipuri-dashboardreporter-app_SAV copy new mahendrapaipuri-dashboardreporter-app in to the plugin folder reboot update to Grafana 11.3.0 Grafana restart permission denied when try to create a report create new service Account Token and put it in permission denied (only one Org)

mahendrapaipuri commented 3 weeks ago

Cheers for quick testing @J-Paul0815

Could you please share the Grafana server logs, if possible in debug mode? When you create service token did you use Admin role for the service account? You can use externalServiceAccounts which will provision the token automatically. However, from Grafana v11.3.0+, you need to enable them explicitly using following config:

[auth]
managed_service_accounts_enabled = true

Could you give it a try with this as well?

Thanks

J-Paul0815 commented 3 weeks ago

I have to thank you. After I made the [Auth] managed_service_accounts_enabled = true entry (restart), I got a request, but the panel was empty. I set logging to debug [log] level = debug. Here is the log:

root@Influx-Grafana-LXC:/var/log/grafana# tail grafana.log
logger=plugin.grafana-image-renderer t=2024-10-29T18:12:27.041089708Z level=debug msg="Using browser version" browserVersion=HeadlessChrome/120.0.6075.0
logger=plugin.grafana-image-renderer t=2024-10-29T18:12:27.046536044Z level=debug msg="using plugin" version=2
logger=secret.migration t=2024-10-29T18:12:27.051141767Z level=debug msg="Finished secret migration service" service=*migrations.MigrateToPluginService
logger=infra.lockservice t=2024-10-29T18:12:27.051196528Z level=debug msg="Execution finished" actionName="secret migration task " duration=1.906418337s
logger=plugins.update.checker t=2024-10-29T18:12:27.052431266Z level=debug msg="Checking for plugin updates" url="https://grafana.com/api/plugins/versioncheck?grafanaVersion=11.3.0&slugIn=mahendrapaipuri-dashboardreporter-app%2Cspectraphilic-windrose-panel%2Cgrafana-image-renderer%2Cgrafana-lokiexplore-app"
logger=infra.lockservice t=2024-10-29T18:12:27.066969397Z level=debug msg="LockExecuteAndRelease finished" actionName="secret migration task " duration=1.931444092s
logger=server t=2024-10-29T18:12:27.067032751Z level=debug msg="Stopped background service" service=*migrations.SecretMigrationProviderImpl reason=null
logger=id-service t=2024-10-29T18:12:27.069882768Z level=debug msg="Cached token found" id=user:1
logger=plugins.update.checker t=2024-10-29T18:12:27.105055512Z level=info msg="Update check succeeded" duration=1.940215191s
logger=ngalert.scheduler t=2024-10-29T18:12:30.000901023Z level=debug msg="Alert rules fetched" rulesCount=0 foldersCount=0 updatedRules=0
root@Influx-Grafana-LXC:/var/log/grafana# tail grafana.log
logger=id-service t=2024-10-29T18:12:58.815882018Z level=debug msg="Cached token found" id=user:4
logger=accesscontrol t=2024-10-29T18:12:58.81657792Z level=debug msg="Evaluating permissions" id=user:4 orgID=1 permissions="action:plugins.app:access scopes:plugins:id:grafana-lokiexplore-app"
logger=accesscontrol.evaluator t=2024-10-29T18:12:58.817349816Z level=debug msg="Matched scope" userscope=plugins:* targetscope=plugins:id:grafana-lokiexplore-app
logger=id-service t=2024-10-29T18:12:58.997306961Z level=debug msg="Cached token found" id=user:4
logger=accesscontrol t=2024-10-29T18:12:58.99798491Z level=debug msg="Evaluating permissions" id=user:4 orgID=1 permissions="action:alert.rules:read scopes:"
logger=ngalert.api t=2024-10-29T18:12:58.999988384Z level=debug msg="User does not have access to any namespaces"
logger=live t=2024-10-29T18:12:59.794911519Z level=debug msg="Client disconnected" user=4 client=82523b23-3bca-4a7b-b242-9aaaac86d666 reason="connection closed" elapsed=1.346267878s
logger=ngalert.scheduler t=2024-10-29T18:13:00.006844061Z level=debug msg="Alert rules fetched" rulesCount=0 foldersCount=0 updatedRules=0
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-10-29T18:13:00.066429132Z level=info msg="report generated" dash_uid=ce0v45vmothxca endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app user=admin
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-10-29T18:13:00.066556313Z level=debug msg="Plugin Request Completed" pluginID=mahendrapaipuri-dashboardreporter-app status=ok statusSource=plugin duration=6.695958186s endpoint=callResource

image

mahendrapaipuri commented 3 weeks ago

Thanks again @J-Paul0815 for your quick responses. Really appreciate it!!

Could you share the full logs please? Right from where plugin is loaded.

Thanks!!

J-Paul0815 commented 3 weeks ago

Thanks again @J-Paul0815 for your quick responses. Really appreciate it!!

Could you share the full logs please? Right from where plugin is loaded.

Thanks!! Sure, you always answer very quickly, thank you! Is this what you need?

grafana.log

mahendrapaipuri commented 3 weeks ago

@J-Paul0815 I see the following log lines.

logger=dashboard.permissions t=2024-10-29T18:12:46.463809542Z level=debug msg="Access denied to dashboard" identity=service-account:4 id=1 permissions="action:dashboards:write scopes:dashboards:uid:ce0v45vmothxca"

Probably the service token that you have created does not have enough permissions. When both manually created service token and service token created by externalServiceAccounts exists, the plugin prioritizes the manually created one. Here are the logs that confirm it:

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-10-29T18:29:48.48843349Z level=debug msg="using user configured token" endpoint=callResource pluginID=mahendrapaipuri-dashboardreporter-app

Please do the following:

[auth]
managed_service_accounts_enabled = true

[feature_toggles]
enable = accessControlOnCall,idForwarding,externalServiceAccounts

You should see a service account created for the plugin as below:

image

You should see the following permissions for the plugin when you go to http://<your_grafana>/plugins/mahendrapaipuri-dashboardreporter-app?page=iam:

image

J-Paul0815 commented 3 weeks ago

I've tried many times, but unfortunately without success. Maybe it's too late, I'll try again tomorrow morning.

J-Paul0815 commented 3 weeks ago

Remove the external service account and the token you have configured for the plugin. Yes. Ensure following config exists in your grafana.ini [auth] managed_service_accounts_enabled = true

[feature_toggles] enable = accessControlOnCall,idForwarding,externalServiceAccounts Yes Restart Grafana server and attempt to generate the report. Yes image image image grafana.log

mahendrapaipuri commented 3 weeks ago

Hey @J-Paul0815 Thanks again for the screenshots and logs. There are few issues here:

logger=plugins.backend.start t=2024-10-29T20:16:01.376628269Z level=error msg="Could not start plugin backend" pluginId=mahendrapaipuri-dashboardreporter-app error="fork/exec /var/lib/grafana/plugins/mahendrapaipuri-dashboardreporter-app/gpx_dashboardreporter-app_linux_amd64: permission denied"
logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-10-29T20:16:01.409149508Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/mahendrapaipuri-dashboardreporter-app_SAV/gpx_dashboardreporter-app_linux_amd64 args=[/var/lib/grafana/plugins/mahendrapaipuri-dashboardreporter-app_SAV/gpx_dashboardreporter-app_linux_amd64]

Restart the Grafana server and try again. If it does not work, please share a scraanshot of plugin config as well along with logs. I hope this time it will work.

J-Paul0815 commented 3 weeks ago

Thanks for your help chmod +x /var/lib/grafana/plugins/mahendrapaipuri-dashboardreporter-app/gpx_dashboardreporter-app_linux_amd64 Yes remove folder /var/lib/grafana/plugins/mahendrapaipuri-dashboardreporter-app_SAV Yes Restart the Grafana server Permission denied grafana.log

image image image

I´ll will be look at it, tomorrow morning again.

wulfchr commented 3 weeks ago

Hello @mahendrapaipuri, I too deleted the old plugin and installed the new plugin. The permissions for the files are set correctly and the parameters are also set correctly in the Grafana default.ini. unfortunately without success. A few pictures and the log file are attached. Thank you very much for your help. api_plugins_mahendrapaipuri-dashboardreporter-app_resources_repo Configuration - Grafana Dashboard Reporter - Plug-ins - Plugins und Daten - Verw grafana.log plugin-file-permissions

mahendrapaipuri commented 3 weeks ago

Thanks a lot @wulfchr @J-Paul0815 for the tests. I see the issues both facing are due to invalid or stale tokens. Please try doing following steps to clean up stale configs and tokens:

After the Grafana server has restarted, verify the following:

Now try to generate a report. If things still do not work, please share the screenshots of plugin config and IAM and service accounts page along with Grafana logs.

Hopefully this time it will work! Fingers crossed.

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri Thank you for your help and especially for your patience. I tried so many things yesterday, not everything was right, let me explain: From plugin configuration page, Reset the Service Account Token and Save Settings. This will clean up any manually configured tokens Ok, no Problem Go to Service Accounts page and delete the service account created by externalServiceAccounts feature flag. The name of that service account will be extsvc-mahendrapaipuri-dashboardreporter-app. This is not possible image So I set: [auth] managed_service_accounts_enabled = false restart Grafana Now I can delete Service Account Token image restart Grafana image So I set: [auth] managed_service_accounts_enabled = true restart Grafana image No Token! image No Permission

mahendrapaipuri commented 3 weeks ago

@J-Paul0815 If the data on this test instance is not very important, I would suggest you to delete the Grafana DB at /var/lib/grafana/grafana.db and restart the Grafana. This will ensure you start clean.

It is very wierd that Grafana creates a service account for plugin but not create a token. I have never seen it. Could you share the logs please?

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri After I deleted the DB, I first had to restore the data source to InfluxDB. The service account now also had a token, but a PDF report could not be created "error generating report". Attached is the LOG. But allow me to ask a completely different question: The HTTP request is only for testing for me, the actual use is (automatically) via "curl" with tokens. You said that manually created tokens are primarily used, if available. Will it then be possible to use it via "curl" in the future? grafana.log

mahendrapaipuri commented 3 weeks ago

Thanks for the test @J-Paul0815

But allow me to ask a completely different question: The HTTP request is only for testing for me, the actual use is (automatically) via "curl" with tokens. You said that manually created tokens are primarily used, if available. Will it then be possible to use it via "curl" in the future?

Yes, it is possible to generate reports using API. Check this section in docs. You need to create a service account and a token and use that token to curl requests.

Between the logs you sent me are incomplete. Could you send me the logs until "error generating report" log line found?

Thanks

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri I can't figure out why something is missing, since I transferred the entire log file via FTP. OK, here's another try. Thank you! grafana.log

mahendrapaipuri commented 3 weeks ago

Cheers @J-Paul0815

Now I see the error log lines and it is coming from grafana-image-renderer.

logger=plugin.grafana-image-renderer t=2024-10-30T13:17:28.431255915Z level=error msg="Error while waiting for the panels to load" url="http://localhost:3000/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000&render=1" err="TimeoutError: Waiting failed: 60000ms exceeded\n    at Timeout.<anonymous> (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/common/WaitTask.js:59:37)\n    at listOnTimeout (node:internal/timers:564:17)\n    at process.processTimers (node:internal/timers:507:7)"

logger=plugin.mahendrapaipuri-dashboardreporter-app t=2024-10-30T14:18:10.812842245Z level=error msg="error generating report" endpoint=callResource err="error rendering PNGs in parallel for dashboard Erreichbarkeit: error rendering PNG: error getting panel : error executing request for http://localhost:3000/render/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000: Get \"http://localhost:3000/render/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000\": downstream error: net/http: timeout awaiting response headers" pluginId=mahendrapaipuri-dashboardreporter-app

What happens when you visit the URL http://localhost:3000/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000&render=1 in your browser?

These sort of transient errors can happen due to puppeteer which grafana-image-renderer uses. Try to restart Grafana server and try again?

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri Thanks! I´ll changed localhost to the Ip-Adress from Grafana, and: image

mahendrapaipuri commented 3 weeks ago

And what happens when you restart Grafana and try again? Is it possible to share your dashboard JSON?

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri For me, rendering works as expected, even after a reboot, but the report is not generated "error generating report". Attached is the panel JSON. Panel_JSON.txt

mahendrapaipuri commented 3 weeks ago

As a last try, can you add this dashboard to your Grafana and attempt to generate a report?

mahendrapaipuri commented 3 weeks ago

And what happens when you visit this URL http://localhost:3000/render/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000&render=1 in your browser?

J-Paul0815 commented 3 weeks ago

As a last try, can you add this dashboard to your Grafana and attempt to generate a report?

Import no Problem Generate Report: error generating report

J-Paul0815 commented 3 weeks ago

And what happens when you visit this URL http://localhost:3000/render/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000&render=1 in your browser?

image

And also rendering from your demo Panel: image

But it needs time (>1 min.)

mahendrapaipuri commented 3 weeks ago

But it needs time (>1 min.)

That is some encouraging news. Because the errors I saw from your earlier logs are timeout errors

logger=plugin.grafana-image-renderer t=2024-10-30T13:17:28.431255915Z level=error msg="Error while waiting for the panels to load" url="http://localhost:3000/d-solo/de2fc2libiuwwb/_?from=now-1h&height=500&panelId=panel-1&theme=light&to=now&width=1000&render=1" err="TimeoutError: Waiting failed: 60000ms exceeded\n    at Timeout.<anonymous> (/snapshot/src/node_modules/puppeteer-core/lib/cjs/puppeteer/common/WaitTask.js:59:37)\n    at listOnTimeout (node:internal/timers:564:17)\n    at process.processTimers (node:internal/timers:507:7)"

It says that grafana-image-renderer waited for 60s for before giving up. It is very wierd that it takes such a long time to render. How big is the server you are using to test this?

Look at the docs on how to increase timeout.

J-Paul0815 commented 3 weeks ago

That is some encouraging news. Because the errors I saw from your earlier logs are timeout errors

It´s an LXC Container on Proxmox, Intel NUC i3 32 GB RAM, the LXC Container 2 Cores, 2 GB, thats what I normaly use. Before I start the upgrades rendering needs 1-3 sec. But OK, I´ll try with 4 Cores and 6 GB, but there is no difference.

If you don't mind, I'll go back with a rollback, there's not much more we can do here now, right?

mahendrapaipuri commented 3 weeks ago

I am out of ideas to be honest. Yes, you can roll back.

If you really want to see plugin in action, clone the repo and do a docker-compose up in the root of the repo. You will see everything works as expected.

mahendrapaipuri commented 3 weeks ago

Thanks a lot for the patient testing @J-Paul0815

Anyways the errors you are having are not from current plugin. So, I guess the problem lies elsewhere!

J-Paul0815 commented 3 weeks ago

Thanks a lot for the patient testing @J-Paul0815

Anyways the errors you are having are not from current plugin. So, I guess the problem lies elsewhere! @mahendrapaipuri I can't thank you enough for your effort, I'm very grateful for it. I'm now back on Grafana 11.2.2. Rendering takes less than 5 seconds, a PDF report takes a little longer, but it works. I have no doubt that the PDF reporter works in general, I'll definitely try the upgrade again later. Thanks again!

mahendrapaipuri commented 3 weeks ago

I'm now back on Grafana 11.2.2. Rendering takes less than 5 seconds, a PDF report takes a little longer, but it works.

Thats very valuable information. So, the issue lies with Grafana 11.3.0 then. Good to know that!!

J-Paul0815 commented 3 weeks ago

I'm now back on Grafana 11.2.2. Rendering takes less than 5 seconds, a PDF report takes a little longer, but it works.

Thats very valuable information. So, the issue lies with Grafana 11.3.0 then. Good to know that!!

I have no idea if that could be the reason, but that could be different: https://www.laub-home.de/wiki/Grafana_Verbindung_zu_InfluxDB_v2_mit_InfluxQL

Yes, it is in German, but it should be recognizable. But actually this should only affect the connection between InfluxDB and Grafana.

wulfchr commented 3 weeks ago

Hello @mahendrapaipuri, your new plugin together with your tip to recreate the database grafana.db is the solution. I did the following in detail:

  1. Export existing dashboards
  2. Grafana server stopped
  3. Database grafana.db deleted
  4. Grafana server started
  5. Grafana login admin/admin assign new password
  6. your report plugin new activated
  7. New data source (influxdb) set up
  8. the new datasource uid replaced in dashboard JSON files
  9. Dashboards imported
  10. Done
  11. Report generated It works very well!!! (see PDF in the appendix) SHLM-Klimasensor-001.pdf

Thank you very much for your great support

mahendrapaipuri commented 3 weeks ago

@wulfchr Awesome, thanks for the tests and reporting back. Appreciate it!

@J-Paul0815 I dont have experience with InfluxDB but from what I understand from your tests is that the same InfluxDB is working "normally" with Grafana v11.2.2 and have timeout issues on Grafana v11.3.0. That looks like there is some sort of regression in Grafana v11.3.0 that is showing itself in your use case.

Anyways thanks a lot to both of you for the tests and support. I will merge this PR and make a new release.

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri Just for your information, maybe it will help: I updated to 1.7.0 today (thanks) and checked/added the entries in Grafana.ini ([auth] managed_service_accounts_enabled = true). I was able to render and create PDF reports. Then I updated Grafana and the problems started: The first thing I noticed was that the Share/Direct Render Link did not have the Grafana IP address, but localhost. If I swapped localhost for the IP address, it was rendered, but it took an extremely long time. I was also no longer able to create a report. I suspect a bug in Grafana version 11.3.0 Github Issue Link As I said, just for your information. Thank you for your work.

mahendrapaipuri commented 3 weeks ago

@J-Paul0815 Cheers for the issue link. So, it was indeed a performance regression from Grafana then.

Anyways, if you have configured the Grafana server to bind to a specific IP address other than localhost, you will have to configure the plugin as well to use that IP address. Please check the appUrl in plugin configuration.

J-Paul0815 commented 3 weeks ago

@mahendrapaipuri I am on Grafana v11.3.0: Rendering works (as fast as usual), PDF Reprort works Here is the solution to the puzzle: Short: You have to update the rendering plugin It should be included in the documentation.

Long: Rendering Plugin Version: image grafana-cli plugins install grafana-image-renderer Grafana restart

image Rendering Plugin Version: image (as user in sudo Group) sudo apt-get install -y adduser libfontconfig1 musl sudo wget https://dl.grafana.com/oss/release/grafana_11.3.0_amd64.deb sudo dpkg -i grafana_11.3.0_amd64.deb image

image

Hope it helps. @wulfchr I don't know what kind of machine you're running on, but it seems to be powerful enough to compensate for the error. An update of the rendering plugin won't hurt you ;-)