Grafana public URL stopped working

[walterzw]

I am using the free tier of Grafana Cloud to retrieve AWS timestream data and to share a public grafana panel with a few selected friends.

This worked well for several weeks, but stopped working a few days ago. Calling the public URL fails now with GET https://*******.grafana.net/api/plugins/grafana-timestream-datasource/settings 401 (Unauthorized)

I blanked my application name in the above URL, but am happy to supply the complete URL directly to someone looking at the problem. If I log in first into grafana cloud, the URL works perfectly well in another tab of the same browser (possibly picking up some access token). Without such login, I get the 401 error followed by a "plugin not found" message.

I opened a ticket with grafana cloud. They can reproduce the error, but point out that the timestream plugin is not listed as supported on https://grafana.com/docs/grafana/latest/dashboards/dashboard-public/#supported-data-sources.

They presume that it stopped working with their recent move to grafana 11. My plugin version is v2.9.1.

Regards WalterZW

[walterzw]

Just got that from grafana cloud: the current version of your hosted Grafana is 11.0.0-67429.

[iwysiu]

Hi @walterzw ! I'm investigating this and I wasn't able to repro it with a basic timestream dashboard. Do you have a link or reference for the Cloud Grafana issue or can you provide the dashboard json (with any sensitive information hidden/removed) of a dashboard with this issue?

[walterzw]

Hi @walterzw ! I'm investigating this and I wasn't able to repro it with a basic timestream dashboard. Do you have a link or reference for the Cloud Grafana issue or can you provide the dashboard json (with any sensitive information hidden/removed) of a dashboard with this issue?

[iwysiu]

Hi @walterzw , thanks! I do see it erroring. I removed the link from your reply since this project is public. Can you give me the dashboard json? (If you click share, then export you can save it to a file). Also, just so I have all the information, are you using assume role to connect or just keys?

[walterzw]

Hi Isabella, the json is attached. When accessing my Grafana instance on Grafana Cloud, I do use userid/password to log in. That user has admin rights and I do configure the panels under that username. On AWS, I do have a specific user that has the necessary roles attached.

[walterzw]

Hi Isabella, just realized something that may be of interest. A few weeks ago, I had to switch my AWS account. When the new account was up and running, I did connect Grafana to the new Timestream instance. I ended up with two connections: “Amazon Timestream” (the original one) and “grafana-timestream-datasource” (the new one). Changing the panels accordingly made the transition painless and all worked as before.

Not sure whether that could be relevant or not.

[iwysiu]

Hi @walterzw ! So to be clear, did updating the panels fix your issue? This may be unnecessary in that case, but I don't think the JSON attached, maybe because you're responding by email.

[walterzw]

Hi Isabella, no, the issue is still present. I did make the change from one timestream source to the other a few weeks ago. At that time, everything continued to work until a week ago (roughly when Grafana Cloud upgraded to grafana 11). I looked through the JSON – no indication that the outdated timestream source is referred to. Only wanted to mention that there was such a switch in case it rings a bell. I attach the JSON here:

WG Rosental Version 0.2-1709653204725.json

[iwysiu]

Ah that's unfortunate 😞 I'll take another look at it. Out of curiosity, if you make a new dashboard (you could just do one panel and not remake the whole thing) does it still have this issue? I'm wondering if it's related to the migration from version 10 to version 11.

[walterzw]

I have done just that and also created a new timestream datasource and linked to that one. Same behavior. Works perfectly fine when logged in first in the same browser, but not with a fresh browser instance. This was done on suggestion by the person who handled my original ticket with Grafana Cloud (https://grafana.com/orgs/quelle/tickets/128200) but pointed out that the timestream plugin is not supported. I have included the link to that ticket to avoid any duplication of efforts.

I guess I could set up a completely new Grafana Cloud account to make sure that every step is done under version 11. Before doing so, what sort of access token is required to read the timestream configuration file that is currently returning the 401 error?

[walterzw]

Just to mention: I created a brand new Grafana Cloud account and linked it to AWS. All done under Grafana 11.

Same behavior - so perhaps a pure Timestream Grafana 11 issue rather than anything linked to the migration to version 11.

[sarahzinger]

Hey there, We think this issue is related to a feature toggle for a new security feature the team has been rolling out: https://github.com/grafana/grafana/blob/2795f9827a8a6e41e6fb6484e999e654fbaa4ea6/docs/sources/setup-grafana/configure-grafana/feature-toggles/index.md?plain=1#L133

@walterzw I believe we disabled this feature on your old account, so the dashboard should be working now. But if you would like it disabled as well on your new account feel free to chat with support and they should be able to help you out! Thanks for reporting the issue, I've let the team working on this new feature know about this edge case so they can disable it for everyone and/or work on a bug fix.

Thanks again for reporting!

[walterzw]

I can confirm, that the public URLs are working again. Also, I deleted the second account as it was only set up for debugging purposes.

MANY THANKS for investigating and fixing it !!!