search

grafana / grafana-operator

An operator for Grafana that installs and manages Grafana instances, Dashboards and Datasources through Kubernetes/OpenShift CRs
https://grafana.github.io/grafana-operator/
Apache License 2.0
863 stars 384 forks source link

[Bug] get version does not work with Azure Managed Grafana #1569

Closed arminfelder closed 2 months ago

arminfelder commented 3 months ago

Describe the bug

The Grafana Operator tries to get the version of the external Grafana instance from its /api/health endpoint, unfortunatly this does not work wit Azure Managed Grafana, as in this case, all endpoints, including the /api/health require authentication.

Secondly the Grafana Operator, seams not to check the HTTP status code (in my case 403) , but only if the request is successful.

Log:

2024-06-05T17:20:23Z    ERROR   GrafanaReconciler       failed to get version from external instance    {"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"external-grafana","namespace":"grafana-operator"}, "namespace": "grafana-operator", "name": "external-grafana", "reconcileID": "79f3e706-0b32-4053-bdac-952e33464692", "error": "parsing health endpoint data: EOF"}
github.com/grafana/grafana-operator/v5/controllers.(*GrafanaReconciler).Reconcile
        github.com/grafana/grafana-operator/v5/controllers/grafana_controller.go:95
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:222

Version 5.9.2 To Reproduce

configure Azure Managed Grafana as external Grafana

Expected behavior A clear and concise description of what you expected to happen.

Suspect component/Location where the bug might be occurring https://github.com/grafana/grafana-operator/blob/f64b54109338478c0cd4a049174584b4e6613d0b/controllers/grafana_controller.go#L157

Runtime (please complete the following information):

theSuess commented 2 months ago

@weisdd will check how authentication for this endpoint is implemented and we'll decide what to do from there

weisdd commented 2 months ago

@theSuess The health endpoint can be accessed using the same API key as for managing grafana instance:

image

So, I think the fix would be to simply use the same client as in various controllers.

arminfelder commented 2 months ago

@theSuess The health endpoint can be accessed using the same API key as for managing grafana instance:

image

So, I think the fix would be to simply use the same client as in various controllers.

this was also me thought, after debugging with curl, but I did not want to mess up your code

DanielAtanasovski commented 2 months ago

Getting a similar issue with AWS Managed Grafana on Version 5.9.2:

2024-06-13T04:33:13Z    ERROR   GrafanaReconciler   failed to get version from external instance    {"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana-workspace","namespace":"grafana"}, "namespace": "grafana", "name": "grafana-workspace", "reconcileID": "e3efad02-5a15-4bc8-a527-2f30cc480e7b", "error": "parsing health endpoint data: invalid character 'B' looking for beginning of value"}
github.com/grafana/grafana-operator/v5/controllers.(*GrafanaReconciler).Reconcile
    github.com/grafana/grafana-operator/v5/controllers/grafana_controller.go:95
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    sigs.k8s.io/controller-runtime@v0.18.3/pkg/internal/controller/controller.go:222
weisdd commented 2 months ago

@DanielAtanasovski I don't have an AWS environment at hand, so could you send an authenticated request (using the same credentials that you supply to the operator) to that endpoint and post the result here?

bonclay7 commented 2 months ago

Is there a possibility to avoid that api endpoint by specifying the grafana version in configs?

DanielAtanasovski commented 2 months ago

@DanielAtanasovski I don't have an AWS environment at hand, so could you send an authenticated request (using the same credentials that you supply to the operator) to that endpoint and post the result here?

It might be worse with AWS as the api/health endpoint appears to be blocked. An authenticated request just responds with Bad Request: Not allowed.

Related to this issue: https://github.com/aws/amazon-managed-grafana-roadmap/issues/41

VermaPriyanka commented 2 months ago

@DanielAtanasovski Amazon Managed grafana supports this API endpoint: https://grafana.com/docs/grafana/latest/developers/http_api/other/#frontend-settings-api. Can this be used alternatively to get the version?