Alert manager: Authorization header is missing

tonypowa commented 4 months ago

What happened?

We use the Prometheus Datasource. We use a custom auth wrapper on top of Prometheus that only accepts oauth2 tokens. When creating an alert in the alert manager. Everything works fine. The token is used for all the dashboards and alert rule preview. When saving the Alert Rule, all rules get the same status: No data. In the backend i see that all requests from alert manager get a 403 with the message: "authorization header is missing"

What did you expect to happen?

That the alert manager backend sends the request with a forwarded OAuth Identity

Did this work before?

I don't know. This is the first time we implement the Alert Manager

How do we reproduce it?

Create a Prometheus datasource
Enable "Forward OAuth Identity"
Create a Alert Rule
Wait a while
Get a NoData label

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 10.3.1

OS: Kubernetes

Browser: Firefox

Grafana platform?

Kubernetes

Datasource(s)?

Latest embedded Prometheus datasource

tonypowa commented 4 months ago

This issue is a test copy of an issue in another repo. Original issue: https://github.com/grafana/grafana/issues/#83365

moxious commented 4 months ago

Summary: User tonypowa reports an issue where the alert manager in Grafana fails to forward the OAuth Identity causing a 403 authorization header is missing error, leading to all alert rules having a status of 'No data' on a Kubernetes environment.

moxious commented 4 months ago

Elaboration:

Hello @tonypowa, thank you for bringing this issue to our attention. It sounds like there might be a misconfiguration with the alert manager's authentication headers. To help us investigate this further and find a resolution, could you please provide the following additional information?

Could you confirm if the OAuth tokens are expected to be passed by a proxy or directly by the alert manager? Any information about your configuration might help.
Are there any logs available from the alert manager or Prometheus that indicate the missing authorization header when the request is made? If possible, please provide the relevant log entries.
Have you made any custom configurations or modifications to the Grafana Alert Manager that might affect the authentication flow?
Is the Forward OAuth Identity configuration working as expected for other use cases outside the alert manager context in your setup?
You've mentioned running Grafana on Kubernetes. Could you share the relevant section of your Kubernetes deployment YAML that pertains to the Grafana or Prometheus service, with any sensitive information redacted?
Lastly, as a temporary workaround, have you tried manually setting the Authorization header or using API tokens instead of OAuth tokens to see if the alert manager accepts the requests?

Any additional context or configuration details you can provide will be immensely helpful in diagnosing the problem. Screenshots of the configurations or a reproduction with a simpler setup (if possible) might also provide valuable clues.

Looking forward to your response so we can help you resolve this issue promptly!

Best regards, [Your Name]

moxious commented 4 months ago

Hello @tonypowa, it sounds like this issue pertains to the handling of OAuth tokens and the Alerting system within Grafana. I believe the most appropriate team to look at this would be the Alerting project. They specialize in the functionality surrounding alert rules and notifications, which appears to be at the core of the problem you're facing with the Alert Manager's authorization mechanism.

moxious / triage