oci_* metric namespaces not fully loading using oci metrics plugin (instance principal auth, single tenancy) v.5.0.0

[riselka]

The dropdown for the metrics namespaces is not loading. We get a couple namespaces in the list but not all.

datasource setup looks good

region list and compartment list load fine, and when inspecting the json the compartment id that is loaded in the display shows the correct compartment ocid in the json.

but the namespace drop down only shows 2 namespaces, it should also show the oci* service metric namespaces:

to make it more confusing, sometimes it doesnt load the 2 in the shot above, and sometimes it shows 1 or 2 of the oci_* namepaces as well...

In the logs we see these errors:

logger=plugin.oci-metrics-datasource t=2024-02-16T18:53:49.640738833Z level=error msg=plugin dsSettings.Environment="dsSettings.Environment: OCI Instance"
logger=plugin.oci-metrics-datasource t=2024-02-16T18:53:49.640757248Z level=error msg=plugin dsSettings.TenancyMode="dsSettings.TenancyMode: "
logger=plugin.oci-metrics-datasource t=2024-02-16T18:53:49.80397137Z level=error msg=plugin CheckHealth=oci-metrics-datasource
logger=plugin.oci-metrics-datasource t=2024-02-16T18:53:49.804023028Z level=error msg=client TestConnectivity="testing the OCI connectivity"
logger=plugin.oci-metrics-datasource t=2024-02-16T18:53:49.804050771Z level=error msg=TestConnectivity TestingTenancyOCID=ocid1.tenancy.oc1..aaaaaaaa22uqqkiuq2qb6r3vzjm26ckitmi6fo4krkle7ymuw2lq26yingea ConfigKey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-16T18:53:49.877005291Z level=error msg=TestConnectivity ConfigKey=DEFAULT/ OK=200
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.13039155Z level=error msg=client GetSubscribedRegions="fetching the subscribed region for tenancy: "
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.131000184Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.131017216Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for tenancy OCID: ocid1.tenancy.oc1..aaaaaaaa22uqqkiuq2qb6r3vzjm26ckitmi6fo4krkle7ymuw2lq26yingea"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.379083841Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for regioname: eu-amsterdam-1"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.3791148Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for regioname: eu-frankfurt-1"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.379126703Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for regioname: us-ashburn-1"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:18.3791513Z level=error msg=client GetSubscribedRegionstakey="fetching the subscribed region for regioname: us-phoenix-1"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:21.751185234Z level=error msg=client GetCompartments="fetching the sub-compartments for tenancy: "
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:21.751234917Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:29.09180986Z level=error msg=client GetNamespaceWithMetricNames="fetching the metric names along with namespaces under compartment: ocid1.compartment.oc1..aaaaaaaasfc6m6dcvnizjtbnm4fod6arxexsheln4vuzxx3almbliksg3nxq"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:29.091897367Z level=error msg=GetTenancyAccessKey Validtakey=DEFAULT/
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:29.091922806Z level=error msg=client.utils listMetricsMetadataPerRegion="Data fetch start by calling list metrics API for a particular regions"
logger=plugin.oci-metrics-datasource t=2024-02-16T22:09:29.878186356Z level=error msg=client.utils listMetrics="Error returned by Monitoring Service. Http Status Code: 500. Error Code: InternalError. Opc request id: fbcdacef25f6605559a21e88501df9f4/F3E9FF3B157FD867E94BB90BB6201CC4/58A36ADAF35416B9EF8749394749EBDD. Message: Unexpected error. Please retry with backoff.\nOperation Name: ListMetrics\nTimestamp: 2024-02-16 22:09:29 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.us-ashburn-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.compartment.oc1..aaaaaaaasfc6m6dcvnizjtbnm4fod6arxexsheln4vuzxx3almbliksg3nxq&compartmentIdInSubtree=false&page=d43976557880aacd%3A0\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_500__500_internalerror for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message."
logger=plugin.oci-metrics-datasource t=2024-02-16T22:10:05.838958416Z level=error msg=client GetNamespaceWithMetricNames="fetching the metric names along with namespaces under compartment: ocid1.compartment.oc1..aaaaaaaasfc6m6dcvnizjtbnm4fod6arxexsheln4vuzxx3almbliksg3nxq"

[riselka]

using the metrics explorer on the console shows what the plugin should be listing here:

[riselka]

I have confirmation there is an issue on the Oracle Telemetry (Metrics) side. They've rolled a fix out to FRA region and it is working there for me now (eu-frankfurt-1). They are getting approvals for rolling the fix out to other regions now.

[mamorett]

very good. I will close the issue then.