search

oracle / oci-grafana-metrics

Grafana datasource plugin for OCI metrics
https://grafana.com/grafana/plugins/oci-metrics-datasource
Universal Permissive License v1.0
80 stars 40 forks source link

Plugin Health Check Error when the region ap-hobsonville-1 is selected #301

Open buddikaMSD opened 5 days ago

buddikaMSD commented 5 days ago

When the region is selected as ap-hobsonville-1 I get an error "plugin health check failed. However other OCI regions are working fine. Screenshots below. image

OCI ap-sydney-1 region is working fine image

buddikaMSD commented 5 days ago

The error log shows the message extracted below. Oct 14 22:31:34 wdmzs006.ssi.govt.nz grafana[65050]: logger=plugin.oci-metrics-datasource t=2024-10-14T22:31:34.881662417+13:00 level=error msg="panic triggered" error="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 345 [running]:\nruntime/debug.Stack()\n\t/opt/homebrew/Cellar/go/1.22.5/libexec/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/grafana/grafana-plugin-sdk-go/backend.handlePanic({0x10ab940, 0x1d5e860})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/serve.go:115 +0x25\ngithub.com/grafana/grafana-plugin-sdk-go/backend.defaultGRPCMiddlewares.WithRecoveryHandler.func4.1({0x0?, 0xe3?}, {0x10ab940?, 0x1d5e860?})\n\t/Users/mamorett/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/options.go:36 +0x27\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.recoverFrom({0x1416460?, 0xc009960ae0?}, {0x10ab940?, 0x1d5e860?}, 0x43d869?)\n\t/Users/mamorett/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:54 +0xea\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1.1()\n\t/Users/mamorett/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:30 +0x67\npanic({0x10ab940?, 0x1d5e860?})\n\t/opt/homebrew/Cellar/go/1.22.5/libexec/src/runtime/panic.go:770 +0x132\ngithub.com/oracle/oci-grafana-metrics/pkg/plugin.(OCIDatasource).TestConnectivity(0xc00c081440, {0x1416460, 0xc009960db0})\n\t/lazio/dev/release/oci-grafana-metrics/pkg/plugin/metrics_functions.go:60 +0x4d0\ngithub.com/oracle/oci-grafana-metrics/pkg/plugin.(OCIDatasource).CheckHealth(0xc00c081440, {0x1416460, 0xc009960db0}, 0xc0004be700)\n\t/lazio/dev/release/oci-grafana-metrics/pkg/plugin/plugin.go:199 +0xe5\ngithub.com/grafana/grafana-plugin-sdk-go/internal/automanagement.(Manager).CheckHealth(0x1416460?, {0x1416460, 0xc009960db0}, 0xc0004be700)\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/internal/automanagement/manager.go:60 +0x1fa\ngithub.com/grafana/grafana-plugin-sdk-go/backend.(diagnosticsSDKAdapter).CheckHealth.func1({0x1416460, 0xc009960d50})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/diagnostics_adapter.go:56 +0x7e\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.errorWrapper.func1({0x1416460, 0xc009960d50})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/adapter_utils.go:49 +0x2b\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.logWrapper.func2({0x1416460, 0xc009960d50})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/adapter_utils.go:123 +0x6a\ngithub.com/grafana/grafana-plugin-sdk-go/backend.metricWrapper.func2({0x1416460, 0xc009960d50})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/adapter_utils.go:76 +0x90\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler.tracingWrapper.func3({0x1416460, 0xc009960d20})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/adapter_utils.go:105 +0x802\ngithub.com/grafana/grafana-plugin-sdk-go/backend.wrapHandler({0x1416460?, 0xc009960ba0?}, {0x1, {0xc0000435c0, 0x16}, {0xc006928948, 0x5}, 0xc00c081340, 0x0, 0xc00995a9c0, ...}, ...)\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/adapter_utils.go:33 +0x146\ngithub.com/grafana/grafana-plugin-sdk-go/backend.(diagnosticsSDKAdapter).CheckHealth(0xc00002c580, {0x1416460?, 0xc009960ae0?}, 0xc00c0812c0)\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/diagnostics_adapter.go:53 +0x24e\ngithub.com/grafana/grafana-plugin-sdk-go/backend/grpcplugin.(diagnosticsGRPCServer).CheckHealth(0x1d62620?, {0x1416460?, 0xc009960ae0?}, 0x3?)\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/backend/grpcplugin/grpc_diagnostics.go:53 +0x25\ngithub.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Diagnostics_CheckHealth_Handler.func1({0x1416460?, 0xc009960ae0?}, {0x1175c00?, 0xc00c0812c0?})\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/genproto/pluginv2/backend_grpc.pb.go:319 +0xcb\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1({0x1416460?, 0xc009960ae0?}, {0x1175c00?, 0xc00c0812c0?}, 0xc000266ba0?, 0x12595fc?)\n\t/Users/mamorett/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/recovery/interceptors.go:34 +0x9c\ngoogle.golang.org/grpc.getChainUnaryHandler.func1({0x1416460, 0xc009960ae0}, {0x1175c00, 0xc00c0812c0})\n\t/Users/mamorett/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1211 +0xb2\ngithub.com/grafana/grafana-plugin-sdk-go/backend.defaultGRPCMiddlewares.(ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func9({0x1416460, 0xc009960ae0}, {0x1175c00, 0xc00c0812c0}, 0xc00002d320?, 0xc00c081300)\n\t/Users/mamorett/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:22 +0x28a\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x1416460, 0xc009960ae0}, {0x1175c00, 0xc00c0812c0}, 0xc00002d320, 0x78?)\n\t/Users/mamorett/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1202 +0x85\ngithub.com/grafana/grafana-plugin-sdk-go/genproto/pluginv2._Diagnostics_CheckHealth_Handler({0x10bc200, 0xc000262730}, {0x1416460, 0xc009960ae0}, 0xc0068af400, 0xc00002c920)\n\t/Users/mamorett/go/pkg/mod/github.com/grafana/grafana-plugin-sdk-go@v0.250.0/genproto/pluginv2/backend_grpc.pb.go:321 +0x143\ngoogle.golang.org/grpc.(Server).processUnaryRPC(0xc0005d8200, {0x1416460, 0xc0099609f0}, {0x141dda0, 0xc00023c4e0}, 0xc00692b7a0, 0xc000267230, 0x1d705c0, 0x0)\n\t/Users/mamorett/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1393 +0xe1a\ngoogle.golang.org/grpc.(Server).handleStream(0xc0005d8200, {0x141dda0, 0xc00023c4e0}, 0xc00692b7a0)\n\t/Users/mamorett/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1804 +0xe8b\ngoogle.golang.org/grpc.(Server).serveStreams.func2.1()\n\t/Users/mamorett/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1029 +0x8b\ncreated by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 87\n\t/Users/mamorett/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1040 +0x125\n"

mamorett commented 5 days ago

We are not managing regions in different way. So if one region is not working,can be related to some authentication issue. Did you check your api key ? Did you try maybe to configure oci cli using that key to see if oci cli works ?

buddikaMSD commented 1 day ago

It turns out that there is an environment parameter that needs to be set as follows. OCI_DEFAULT_REALM=sovereigncloud.nz setting this environment in the /usr/lib/systemd/system/grafana-server.service meant that the Grafana OCI datasource for the sovereign cloud is able to connect. However, while I am able to connect and read the compartment I get a message "listMetrics failed in each Compartments in profile DEFAULT/"

Tried out the oci cli to list the monitoring metrics and that fails too. The user that owns the API has the necessary policies set to read the compartments and read metrics.

I have logged a call with Oracle support as there must be something different in the sovereigncloud setup of our compartment.