Not able to connect to datasource

[Kmdkca]

Hi

Running Grafana 10.0.2 in my docker desktop, i try to connect to our OCI tenant. I've tested with a API key that are oci admin and still the plugin states "Data source is not working" Better error messages would be nice, but how do i troubleshoot this?

[mamorett]

Hi. Can you share logs coming from your container when trying to connect ? Thanks.

[Kmdkca]

Configuration:

Log: 2023-07-13 10:58:55 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:55.613133057Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:55 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:55.613422132Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:55 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:55.613499302Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=1 duration=1.991377ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:55 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:55.78661019Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:55 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:55.786711803Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:55 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:55.786747604Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=1 duration=1.389796ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:56 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:56.1943837Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:56 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:56.194487948Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:56 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:56.194596509Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=2 duration=2.879643ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:56 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:56.646698757Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:56 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:56.6469136Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:56 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:56.647004795Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=3 duration=3.255725ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:57 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:57.081793557Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:57 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:57.081926367Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:57 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:57.082023073Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=2 duration=2.875247ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:58 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:58.063855059Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:58 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:58.064180012Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:58 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:58.064314788Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=3 duration=3.24035ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:58 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:58.747090823Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:58 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:58.747205665Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:58 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:58.747238466Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=2 duration=2.735327ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:58:59 logger=plugin.oci-metrics-datasource t=2023-07-13T08:58:59.507688094Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:58:59 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:59.508015046Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:58:59 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:58:59.508090344Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=1 duration=1.81637ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:59:00 logger=plugin.oci-metrics-datasource t=2023-07-13T08:59:00.087400057Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:59:00 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:59:00.087812842Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:59:00 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:59:00.087977117Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=3 duration=3.413444ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query 2023-07-13 10:59:01 logger=plugin.oci-metrics-datasource t=2023-07-13T08:59:01.593808136Z level=error msg="Error with config:DEFAULT" 2023-07-13 10:59:01 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:59:01.59398384Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=172.17.0.1 traceID= 2023-07-13 10:59:01 logger=context userId=1 orgId=1 uname=admin t=2023-07-13T08:59:01.5940463Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=172.17.0.1 time_ms=11 duration=11.496599ms size=116 referer=http://localhost:3000/connections/datasources/edit/d372cfcb-5c6c-4b94-9109-818a4d039432 handler=/api/ds/query

[mamorett]

Hi. Looks like is not able to initiate the communication with the OCI API. This can usually be due to a misconfigured parameter (the "usual" suspect in this case is the API Key, make sure is the one downloaded from OCI console, not modified) or a network communication issue (make sure your container can access the internet and in particular the *oracle.com network)

[Kmdkca]

Hi.

The container is able to access internet. I've downloaded the plugin. Docker host is my pc, so i can sign in to OCI. Would be nice if the test, could verify access to oracle.com first and return message about that instead of having an errormessage that doesn't give you a clue about whats wrong. The API key is the "API fingerprint key", in format bf:e9:11:5d:d7:12:xxxxxx, right? I tried to re-enter everything. still same error

[mamorett]

Yes the API key is in that format. Is the pem key you are using containing exactly 27 lines ? and begins with -----BEGIN PRIVATE KEY----- and ends with -----END PRIVATE KEY----- ? can you also verify the newline of the pem is UNIX format (not Windows) ?

[Kmdkca]

Well, i did test it both with and without the begin/end lines. Without, i get the "Data source is not working" error. With, i get : "Cannot read properties of undefined (reading 'details')"

I downloaded the PEM file (private key) from OCI when creating the API token, but it contains 28 lines (including begin/end)

Th file is UTF8, but how do i verify the format. I'm on windows OS

[mamorett]

I do not know Windows, sorry. With UNIX/Linux OS you can use the command "file" to determine the line end format. However, you copy/pasted this key in the "Private Key" field, right ?

[Tomson124]

Hey, I do have the exact same problem:

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:32.44152414Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:32.441722836Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:32.44179929Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.252837ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:32.956951635Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:32.957136678Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:32.957210387Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.102871ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:33.510568401Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:33.510732939Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:33.510804324Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.115372ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:33.990380103Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:33.990560274Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:33.990634649Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.08875ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:35.695729409Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:35.695928192Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:35.696001731Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.178451ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:36.734521685Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:36.734702349Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:36.734777698Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=16 duration=16.204181ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:37.726303653Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:37.726510591Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:37.726583205Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.197841ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:38.734039676Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:38.734441619Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:38.734522622Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.856838ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=plugin.oci-metrics-datasource t=2023-07-24T16:24:39.740146688Z level=error msg="Error with config:DEFAULT"

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:39.740311168Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = broken environment" remote_addr=192.168.180.42 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-07-24T16:24:39.740377028Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.180.42 time_ms=2 duration=2.118898ms size=116 referer=http://192.168.180.175:3000/connections/datasources/edit/b6c07fc9-5c3f-4e6c-95a5-4e8012075760/ handler=/api/ds/query

logger=cleanup t=2023-07-24T16:27:58.60708791Z level=info msg="Completed cleanup jobs" duration=1.804088ms

Did configure it exactly as the guide said. But Grafana just states "Data Source not working" I am also running Grafana locally but on my NAS.

[mamorett]

hi @Tomson124 : which version of Grafana are you running ? I will try to reproduce the issue in my lab.

[mamorett]

Also, can you confirm you are using the very latest version (4.0.1) of the plugin ? thanks.

[Tomson124]

Yes, I am using OCI 4.0.1. Grafana version 10.0.2 Probably should also add Grafana runs in a Docker container on my QNAP NAS (linux based). It can access the internet, verified by pinging different adresses from inside the container.

[mamorett]

Hi guys. I was able to reproduce the issue. This is due to recent changes in Grafana 10 and limited support for AngularJS. A couple of things here. you should put in your grafana.ini config file this statement: " angular_support_enabled = true". With that I still got errors while configuring the datasource, however the datasource seems to work (I can create dashboards, etc). I will open a bug and we will start to work for a bugfix. Meanwhile you can try as suggested to set the angular support and try to create a sample dashboard ignoring the error in datasource configuration. If this will not work I think, for now, you can use a different version of Grafana (v9 or v8). I will update on progresses on the bugfix.

[Tomson124]

I think in my case the problem is something else. I set up a new container wit Grafana v9.5.6 and still get the same error and log output. Could it be a new limitation to the Oracle free tier? But then again it did work in the past.

[mamorett]

I tried with Grafana 9.5.6 and it works without errors. and yeah, I tested using a free account too.

[michaeloa]

Any word on when a fix for this will be released? We have been seeing the same issue with our Grafana 10.0.2 instance, and had to downgrade to 9.5.8 to get OCI metrics to work.

[mluitens]

Same for me. On my side Granaf 8.x and latest plugin are not working ("Data source is not working")

[mamorett]

Hi all. "Datasource not working" is driving to some misconfiguration in your parameter (as said, the usual suspect is the API key). Current version of the plugin should work till version 9.x, if something is not working till this version then I would recommend to double check the parameters and eventually the policies in your tenancy. It can be useful to try to check on the same host where your Grafana is installed, also to check access using the OCI CLI to test if the service accountis working as expected. User who are using Grafana 10.x may have different problem ("Cannot read properties of undefined") while configuring the datasource. This is due to the deprecation of AngularJS in Grafana 10 and will be fixed in the next plugin release. However if you encounter that exact error, you can ignore for now and try to create a dashboard. As said this is due to AngularJS but does not prevent the current version of the plugin to work with Grafana 10.x too.

[mluitens]

Hi all. "Datasource not working" is driving to some misconfiguration in your parameter (as said, the usual suspect is the API key). Current version of the plugin should work till version 9.x, if something is not working till this version then I would recommend to double check the parameters and eventually the policies in your tenancy. It can be useful to try to check on the same host where your Grafana is installed, also to check access using the OCI CLI to test if the service accountis working as expected. User who are using Grafana 10.x may have different problem ("Cannot read properties of undefined") while configuring the datasource. This is due to the deprecation of AngularJS in Grafana 10 and will be fixed in the next plugin release. However if you encounter that exact error, you can ignore for now and try to create a dashboard. As said this is due to AngularJS but does not prevent the current version of the plugin to work with Grafana 10.x too.

HI Mamorett, Tnx. Even if the API key is working on other servers (where we are able to scale, etc.) it can still be the API key?

[mamorett]

Hi. Yes, it can. There can be two potential issues. First, if your API key is working but the service account or user owner of that key does not have the policies to access metrics. Then you would run into that error. Or if you copy/pasted the API key skipping some characters, or with Windows newlines. Then the key is stored but is somewhat corrupted in the plugin.

[michaeloa]

Well, I can only speak to our experiences. The plugin did not work for us in version 10.0.2 - we have existing dashboard and the data source would not configure, and we got no data into the dashboards. Downgrade to 9.5.8, and it works fine (same API keys and everything).

[mluitens]

Strange, I tested everything which was mentioned in this thread, but everything without succes

• Downgrade to Grafana 8, as wel as downgrade / upgrade to Grafana 9.5.8,
• Tried another user (which is definately able to read the metrics, because we use it on Linux prompts to read the output).

But none of them helped me out. Any suggestions? I am currently running Grafana 9.5.8 and still receiving the same warning ( Datasource not working )

[mamorett]

Can you try to upgrade to the latest plugin version (5.0.0) and double check your connection parameters and see if it fixed the issue ?

[michaeloa]

We will try to upgrade this as soon as possible.

[michaeloa]

Late, sorry. Can confirm this works now.