search

grafana / timestream-datasource

Amazon Timestream in Grafana
https://grafana.com/grafana/plugins/grafana-timestream-datasource
Apache License 2.0
23 stars 19 forks source link

Grafana v10 dashboard errors on combining sources from Athena and Timestream #273

Closed jatin039 closed 6 months ago

jatin039 commented 7 months ago

Hello,

We have a self hosted Grafana (v10.2.0) setup that we host and manage ourselves. We recently upgraded to v10 from v9. We use the Athena Plugin (v2.13.3, https://grafana.com/grafana/plugins/grafana-athena-datasource/) as one of our datasources to display data on many of our dashboards. We also use the AWS Timestream plugin.

We have a few dashboards where we display panels that use the AWS Athena and AWS Timestream plugins side-by-side. Recently, we noticed the following error appearing intermittently on several of our dashboard panels

"Status: -1. Message: Query error: -1 Request was aborted"

What we found was that this most commonly appears whenever we introduce a panel which use the Athena plugin to fetch data. When we have a dashboard full of only Timestream panels or Athena panels, we don't notice this issue. Whenever we combine the two, it occurs immediately.

Furthermore, there are no fixed panels on which it occurs. On each refresh this might change. Sometimes, there are no errors (very rare) while sometimes all panels show the error.

Could you please help us identify the cause of this and what we can do to avoid this? We only began seeing this issue after the Grafana upgrade. We also don't see this issue with any of the other plugins that we use, even when we use them within the same dashboard

sarahzinger commented 7 months ago

Hi there! Thank you so much for reporting this bug!

I just tested with 10.2.0 and with the latest versions of timestream and athena and was able to reproduce this easily with any query I make (I just did "show tables"). Then I tried with grafana main (~10.3.x) and it's also broken there. Pretty consistently if I have 2 panels one for athena and one for timestream, one of them will error with "Query Error: -1 Request was aborted"

Initial investigation notes: I see that one of the queries is always cancelled in the network tab, and in grafana's logs I see [plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Canceled desc = context canceled which points to an issue with the plugin, although I'm not sure what yet. Will move this to Next to have someone from our team take a look in the next sprint or two.

jatin039 commented 7 months ago

@sarahzinger : Thank you for looking into this. Would you be able to provide a estimate date on by when we can expect the fix / path forward to unblock this?

katebrenner commented 7 months ago

linking these as they may be related: https://github.com/grafana/athena-datasource/issues/259

iwysiu commented 7 months ago

Hi @jatin039 ! I just picked this up to investigate it. I don't have a fix yet, but as a workaround, you can set the athenaAsyncQueryDataSupport feature toggle to false. (Which worked for me locally). Docs on feature toggles

iwysiu commented 6 months ago

Hi! I just released Athena v2.13.5, which has the fix for this.