0 series returned

[luitens]

Recently upgraded Grafana to v10.0.0 in combination with Oracle Cloud Infrastructure Metrics version 5.0.0. Installation and configuration went fine, and also the datasource is created with "succesfull".

But when I want to create a dashboard, it seems it's getting no data and it says : 0 series returned

I have the policies as described: allow group grafana to read metrics in tenancy allow group grafana to read compartments in tenancy

And with exactly this same user on the same OS, I am able to retrieve metrics via the prompt of the CLI. Also, the query builder is filling the dropdown lists with the Region, Namespace, Metric, etc. etc. But it does not provide any rows/data.

Any tips/ideas?

[mamorett]

That's very strange, since it seems from the dropdowns you are able to list namespaces and metric. Can you try to actually create a dashboard (then not using the Data Explore) and maybe also with a different namespace and use "Time Serie" Visualization ?

[luitens]

Yes, really strange. Even with time-series, same issue. As you can see the dropdowns are filled. And I just tested with the OCI CLI and that perfectly shows everything. In my Grafana logs I see that it's trying to pick the OCID and read the oci_database metrics (which are working from the CLI)

[luitens]

Just did a test with the root compartment, which is working and showing data. Any idea why I don't see data from the underlying compartments?

This rights are in place allow group <> to read metrics in tenancy allow group <> to read compartments in tenancy

[mamorett]

Do you have the following policy: Allow group <> to read compartments in tenancy ?

[luitens]

I made a copy/ paste typo. yes, I have these two: allow group <> to read metrics in tenancy allow group <> to read compartments in tenancy

[luitens]

I can see this in the Grafana log (so it looks it's just returning 0 series but reading the compartments. These compartments are working via oci cli commands)

logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:41.177422684+01:00 level=error msg=client.utils listMetricsMetadataPerRegion="Data fetch start by calling list metrics API for a particular regions" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:44.925680186+01:00 level=error msg=client GetResourceGroups="fetching the resource groups under compartment 'ocid1.compartment.oc1..' for namespace 'oci_database'" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:44.925749248+01:00 level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/ logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:44.925779937+01:00 level=error msg=client.utils listMetricsMetadataPerRegion="Data fetch start by calling list metrics API for a particular regions" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:46.673196664+01:00 level=error msg=client GetResourceGroups="fetching the resource groups under compartment 'ocid1.compartment.oc1..' for namespace 'oci_database'" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:46.673304631+01:00 level=warn msg=client GetResourceGroups="getting the data from cache" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:47.445664832+01:00 level=error msg=plugin QueryData="Oracle Cloud Infrastructure Metrics" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:47.445727372+01:00 level=error msg=plugin.query query="query initiated for A" logger=plugin.oci-metrics-datasource t=2023-11-09T11:44:47.445752431+01:00 level=error msg=client GetMetricDataPoints="fetching the metrics datapoints under compartment 'ocid1.compartment.oc1..' for query 'CpuUtilization[1m].avg()'"

[mamorett]

IT looks fine. Are you sure you have the correct resources in these compartments ? Can you try different namespaces in different compartments to see if you can grab metrics ?

[luitens]

I tried, no success. From the metric explorer in the portal I see this:

But from Grafana (having the same filters) having this:

[mamorett]

Is the user you are connecting to the Oracle Console the same of the api key you are using in Grafana ? Did you try to restart grafana ?

[luitens]

Yes, I am using the same user, on the same OS (restart of Grafana and whole VM had not success)

[mamorett]

I saw a similar behaviour once (being able to list namespaces and metrics, but getting no data) Was when I setup everything correctly but using a wrong user OCID. Obviously I dunno your context here but would recommend to double check all the params. Other thing: you mentioned you were able to get metrics using the cli. Do you have that output and can you confirm the ~/.oci/config parameters are exactly the same used in grafana ?

[luitens]

I found out the issue. It seems no data is handed over since 2 weeks. So the Plugin and the rights are working perfectly fine (If I set the timeline back ~2 weeks, I have data filled in my tables/graphs), but there is an issue with sending the metrics from ExaCC to OCI. I need to dive in that. Tnx for your help so far

[mamorett]

Very good and nice to hear :) I will close the issue then.