search

oracle / oci-grafana-metrics

Grafana datasource plugin for OCI metrics
https://grafana.com/grafana/plugins/oci-metrics-datasource
Universal Permissive License v1.0
80 stars 40 forks source link

Data source is not working with Oracle Cloud Infrastructure for the grafana deployed on OKE #223

Closed kv00797898 closed 7 months ago

kv00797898 commented 1 year ago

I have deployed my grafana on OKE cluster by using the below documentation. Deployment is success and I am able to connect to the grafana. image

I am facing the below error while adding the DATA SOURCE Oracle Cloud Infrastructure Metrics.

image

Here I am sharing the error logs from the grafana kubernetes pod. Please check and help me to resolve the issue.

Error logs from the grafana Kubernetes pod:

logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:51.294591759Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: bbc7703c2c681868a723e85a23d7cf9f/5E3C42CDB97FB0D7B5BD478EC81E56D8/858ECF12E9A5D416B46174527B6275B6. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:51 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:51.294934295Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=75 duration=75.57506ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:52.022168458Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: b2058456cb6d8fc957866830b6324060/F5EE56F678F73CCB8B8A8BE00291E05C/35DB457C6D2C559AD343BB3626AD8017. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:52 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:52.022217661Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=15 duration=15.333345ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:52.900374936Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: b301b5cd1578e0b4b93e5e3adf1225bd/6B3AA4230B8FF393864E0A64D753D8F3/ACA992113C5CBBFF737F98D705904810. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:52 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:52.900411895Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=9 duration=9.124574ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:53.448296311Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: eb8f621b8786fc95c61034a27a8068a1/4418BD9E7CD3116D7A907C71EC1DA8D3/87A6365E43B3F8D1C0DA0CC0EBA1F73D. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:53 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:53.448364169Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=12 duration=12.774952ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:53.737037291Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: 21be36d286c5319f244989e29f037e7e/11CA971D3374702D9BD5A7CB20A32AD6/6825FD695F64B6B2472CD6C1A1F89721. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:53 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:53.737082805Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=9 duration=9.547139ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:54.751434389Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: b89ff136bfabc2db573d510221427d27/AEC65DA7614DF8A8B6BD40B342AA27A3/1418DE911CFBEA95ABD0CA913A5E3DC7. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:54 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:54.751504982Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=12 duration=12.84776ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:55.634023946Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: 2341549c6322fa366f177f9caaae3bad/43567FBEBDB9B685F50AEB9D1A464313/04748B6816357408FB7D73BE699B9D73. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:55 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:55.634062969Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=32 duration=32.192111ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:56.281391913Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: 6932c2607fc19c7cb197c2c41893f921/A4DF9B64BD81E4FF6431CE3AC51FFB27/8257BB01D54B3D3A2A14BA69535F28D6. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:56 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:56.281437689Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=9 duration=9.676854ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:57.172860962Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: 008fbf226c0f9e8592ebb05e61f59db1/7B77BD32C4DFDBAFE3EACC34EA61E066/BA1CA1CD143DEF1A3AE0E0996554701A. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:57 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:57.172906227Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=18 duration=18.42559ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:59.227598936Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Unknown desc = Error returned by Monitoring Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: bb5a0016ffbb022f8148956a25c64818/A7F4FC7B610571EE4AAD6B755EA81058/9A28519BB5876130080DFC64E872C51C. Message: Authorization failed or requested resource not found.\nOperation Name: ListMetrics\nTimestamp: 2023-08-25 07:27:59 +0000 GMT\nClient Version: Oracle-GoSDK/65.23.0\nRequest Endpoint: POST https://telemetry.me-jeddah-1.oraclecloud.com/20180401/metrics/actions/listMetrics?compartmentId=ocid1.tenancy.oc1..aaaaaaaan3et4ej5l24sbsejheik5bv4fjradz7xgqtr5didj5accbdctmbq\nTroubleshooting Tips: See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_404__404_notauthorizedornotfound for more information about resolving this error.\nAlso see https://docs.oracle.com/iaas/api/#/en/monitoring/20180401/Metric/ListMetrics for details on this operation's requirements.\nTo get more info on the failing request, you can set OCI_GO_SDK_DEBUG env var to info or higher level to log the request/response details.\nIf you are unable to resolve this Monitoring issue, please contact Oracle support and provide them this full error message." remote_addr=10.244.6.1 traceID= logger=context userId=1 orgId=1 uname=admin t=2023-08-25T07:27:59.227746043Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.244.6.1 time_ms=10 duration=10.881043ms size=116 referer=http://grafana-metrics.dxt.online/datasources/edit/d3c82457-c7ad-48e2-8074-65effe9f92ee handler=/api/ds/query

mamorett commented 7 months ago

This was an authorization issue, not a plugin issue.