Trying to integrate with Azure Managed Prometheus but getting Unauthorised issue

[KasyapG]

Report

I followed Integrate KEDA with your Azure Kubernetes Service cluster to create scalers.

And while following these we are unable to add the Lable or environments information to Keda.

[attcloud@dev-31467-muelsoft-poc-cmd-vm-01 ~]$ kubectl get pods -n keda NAME READY STATUS RESTARTS AGE keda-admission-58445797d8-jbj47 1/1 Running 0 3d21h keda-metrics-apiserver-674bb75f7-q6vjl 1/1 Running 0 3d21h keda-operator-9b754d5b4-gx8wt 1/1 Running 2 (3h43m ago) 3d20h

As per the instructions in the issue resolution https://github.com/kedacore/keda/issues/5203, it says that Keda should have the below environments, however we have already installed Keda 2.10, so can we add these variables and Label values by updating the Keda-operator pod? If not where exactly these details to be updated once the Keda is already installed.

Below is the scaled object I have tried. Please review and add comments if this requires any changes as well.

              apiVersion: keda.sh/v1alpha1
              kind: TriggerAuthentication
              metadata:
                name: azure-managed-prometheus-trigger-auth
              spec:
                podIdentity:
                    provider: azure-workload
              ----
              apiVersion: keda.sh/v1alpha1
              kind: ScaledObject
              metadata:
                name: azure-managed-prometheus-scaler
                namespace: 60486a51-7039-4f45-b1be-a0069c86f864
              spec:
                scaleTargetRef:
                  name: test-sample-keda-hpa
                minReplicaCount: 1
                maxReplicaCount: 4
                triggers:
                - type: prometheus
                  metadata:
                    serverAddress: prometheus.monitor.azure.com
                    metricName: access_frequency
                    query: sum((nginx_ingress_controller_requests{service="test-sample-keda-hpa"}))
                    threshold: '1'
              #      activationThreshold: '5.5'
                  authenticationRef:
                    name: azure-managed-prometheus-trigger-auth

Expected Behavior

Keda-operator to have the Workload identity details and connection to be established..

Actual Behavior

Getting 410 error and the scaledobject shows unknown value as it could not connect.

Steps to Reproduce the Problem

See https://learn.microsoft.com/en-us/azure/azure-monitor/containers/integrate-keda

Logs from KEDA operator

2024-03-26T15:02:00Z    ERROR   scale_handler   error getting metric for scaler {"scaledObject.Namespace": "60486a51-7039-4f45-b1be-a0069c86f864", "scaledObject.Name": "azure-managed-prometheus-scaler", "scaler": "prometheusScaler", "error": "prometheus query api returned error. status: 401 response: {\"status\":\"error\",\"errorType\":\"Unauthorized\",\"error\":\"Request does not contain an Authorization Header. RequestId: 61c3955156434f37a4f8c787fff6636f\"}"}

KEDA Version

2.10.0

Kubernetes Version

1.27

Platform

Microsoft Azure

Scaler Details

prometheus

Anything else?

No response

[JorTurFer]

Hello! Those envs are automatically set by the workload identity webhook when a pod whose service account has the annotation is scheduled. That's a component that has to be present in your cluster.

Could you confirm that you have installed it? It is done by the --enable-workload-identity argument during cluster update If you prefer to install if by yourself, you can do it following their guide: https://azure.github.io/azure-workload-identity/docs/installation.html

[KasyapG]

@JorTurFer , We have enabled the workload identity and did a restart of Keda operator as well using the document provided, still see 401 unauthorized in the logs to connect to, so could help us understand the what else could be the issue. I also provided the scaledobeject yaml

[JorTurFer]

Oh, I have just noticed that you are using KEDA v2.10.0. Could you confirm this? The support for Azure managed Prometheus was added in v2.11 so maybe you have to upgrade KEDA

[KasyapG]

Hi @JorTurFer ,

As per the Microsoft documentation it says 2.10 is the required version, so we have upgraded to 2.10

Also we tried using bearer token type as authmode in the scaled object (shown below) as per the documentation "https://keda.sh/docs/2.13/scalers/prometheus/", but still facing an unauthorized issue.

Is there anything else should we look in to.

Regards, Kasyap G

[JorTurFer]

Do you still see the same error about missing header? You configuration looks good, so I'd like to know if you still get the same error or it has changed

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.