search

mullerpeter / databricks-grafana

Grafana Databricks integration allowing direct connection to Databricks to query and visualize Databricks data in Grafana.
Apache License 2.0
55 stars 4 forks source link

Getting Template error when i updated to v1.2.3 #73

Closed udiazulayintel closed 3 months ago

udiazulayintel commented 6 months ago

in grafana log, i see many : logger=plugin.mullerpeter-databricks-datasource t=2024-03-19T19:37:06.1928575+02:00 level=info msg="LongToWide conversion error" err="can not convert to wide series, expected long format series input but got not series"

which were not before.

Thanks!

mullerpeter commented 6 months ago

Not sure if that is related to the latest changes, as nothing on that part of the code changed. Does the error disappear if you disable the long to wide conversion in the advanced settings in the query editor?

udiazulayintel commented 6 months ago

i will try it, i also updated to a new grafana version ... thx

udiazulayintel commented 6 months ago

Hi, well, seems like the error is different, related to the SessionHandle ?

logger=context traceID=948f5a5f3a6d40eda6be0d05a2c20f58 userId=5 orgId=1 t=2024-03-21T10:28:51.258006+02:00 level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to call resources: failed to receive call resource response: rpc error: code = Unknown desc = databricks: execution error: failed to execute query: Invalid SessionHandle: SessionHandle [79cd286f-f47a-4552-8e51-5a2ddb441b34]" remote_addr=10.245.179.50 traceID=948f5a5f3a6d40eda6be0d05a2c20f58 logger=context traceID=948f5a5f3a6d40eda6be0d05a2c20f58 userId=5 orgId=1 t=2024-03-21T10:28:51.258006+02:00 level=error msg="Request Completed" method=POST path=/api/datasources/uid/ca81c5a8-e98d-44d8-95ee-a51cb96f5af7/resources/defaults status=500 remote_addr=10.245.179.50 time_ms=1065 duration=1.0659802s size=148 referer="https://grafanamit/d/c05f85d8-d22c-4a37-b967-63eb6baf4f6d/all-labs-trace-poc?orgId=1&refresh=1m" handler=/api/datasources/uid/:uid/resources/*

mullerpeter commented 6 months ago

This is again the same error as in #61, this happens when the Databricks connection has been idle for too long & the session should be reinitiated. With the logic reintroduced by #62 this should be catched, the Databricks session will be refreshed & the query should be run again after the refresh. This works for me with the plugin version 1.2.3 and on Grafana Version 10.4.1

What version of Grafana are you using, then I'll try to reproduce the error

mullerpeter commented 6 months ago

And what kind of compute resources are you using for the connection on the Databricks side? Serverless SQL Warehouse, SQL Warehouse or All purpose compute cluster?

udiazulayintel commented 6 months ago

Thanks! I am using v10.3.3. it is not always happening, as you said, when it is idle for some time - i get it, after refresh all works well... but, still - bugging :-)

mullerpeter commented 3 months ago

Hi @udiazulayintel

v1.2.6 introduces a new mechanism to open the Databricks sql connection. This could potentially fix the issue with the invalid session handle errors, but this is just a guess. Incase you try the new version let me know if that fixes the issue for you

udiazulayintel commented 3 months ago

Looks good, i will monitor it – Thanks!

Thanks, Udi

From: Peter Müller @.> Sent: Friday, 14 June 2024 23:56 To: mullerpeter/databricks-grafana @.> Cc: Azulay, Udi @.>; Mention @.> Subject: Re: [mullerpeter/databricks-grafana] Getting Template error when i updated to v1.2.3 (Issue #73)

Hi @udiazulayintelhttps://github.com/udiazulayintel

v1.2.6https://github.com/mullerpeter/databricks-grafana/releases/tag/v1.2.6 introduces a new mechanism to open the Databricks sql connection. This could potentially fix the issue with the invalid session handle errors, but this is just a guess. Incase you try the new version let me know if that fixes the issue for you

— Reply to this email directly, view it on GitHubhttps://github.com/mullerpeter/databricks-grafana/issues/73#issuecomment-2168751374, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5GKGU5TJGE4F2KV4BMGDT3ZHNKEZAVCNFSM6AAAAABE6BBODWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRYG42TCMZXGQ. You are receiving this because you were mentioned.Message ID: @.**@.>>

Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

mullerpeter commented 3 months ago

@udiazulayintel Did you experience the issue again in the meantime with the new version? Otherwise I would close this issue

udiazulayintel commented 3 months ago

Hi, yes i do... only for first connection

Udi

From: Peter Müller @.> Sent: Friday, June 28, 2024 9:43:42 PM To: mullerpeter/databricks-grafana @.> Cc: Azulay, Udi @.>; Mention @.> Subject: Re: [mullerpeter/databricks-grafana] Getting Template error when i updated to v1.2.3 (Issue #73)

@udiazulayintelhttps://github.com/udiazulayintel Did you experience the issue again in the meantime with the new version? Otherwise I would close this issue

— Reply to this email directly, view it on GitHubhttps://github.com/mullerpeter/databricks-grafana/issues/73#issuecomment-2197450043, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5GKGU6MBO6A3PO2AHGG5WLZJWVF5AVCNFSM6AAAAABE6BBODWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXGQ2TAMBUGM. You are receiving this because you were mentioned.Message ID: @.***>

Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

mullerpeter commented 3 months ago

Ok this is really strange as this "Invalid SessionHandle" error is catched here when executing a query and then a new DB connection is created and the query is retried recursively via the same function. So in theory this error should never make it to the frontend...

The only queries NOT going via this retry function currently, are the ones for the code auto completion. So it could be that these are the queries leading to this error being displayed.

Do you have Code Auto Completion enabled in the Datasource settings? If so, could you disable it and see if the error still occurs?

mullerpeter commented 3 months ago

@udiazulayintel I just checked the logs again and on initial load of any Dashboards which include a Databricks Datasource the SuggestionProvider is initialised, which fires a query to get the default catalog & schema from databricks. As this query was previously not using the logic to reinit the connection on Invalid SessionHandle errors, it would result in the error you are seeing on first connection (or at least after some idle time).

I fixed this with #83 and v1.2.8 should now finally fix your issue

udiazulayintel commented 3 months ago

Hi,

It looks like it is resolved in 1.2.9! Thanks much

Thanks, Udi

From: Peter Müller @.> Sent: Saturday, 29 June 2024 0:01 To: mullerpeter/databricks-grafana @.> Cc: Azulay, Udi @.>; Mention @.> Subject: Re: [mullerpeter/databricks-grafana] Getting Template error when i updated to v1.2.3 (Issue #73)

@udiazulayintelhttps://github.com/udiazulayintel I just checked the logs again and on initial load of any Dashboards which include a Databricks Datasource the SuggestionProvider is initialised, which fires a query to get the default catalog & schema from databricks. As this query was previously not using the logic to reinit the connection on Invalid SessionHandle errors, it would result in the error you are seeing on first connection (or at least after some idle time).

I fixed this with #83https://github.com/mullerpeter/databricks-grafana/pull/83 and v1.2.8 should now finally fix your issue

— Reply to this email directly, view it on GitHubhttps://github.com/mullerpeter/databricks-grafana/issues/73#issuecomment-2197653333, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5GKGU63LWPTPP3UVNSPCEDZJXFINAVCNFSM6AAAAABE6BBODWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXGY2TGMZTGM. You are receiving this because you were mentioned.Message ID: @.**@.>>

Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.