[alertmanager] exposing smtp secret in plain text for multitenant alertmanager

[ToonTijtgat2]

Describe the bug

I pushed the alertmanager configuration via the mimir tool and was checking the config in grafana for the multitenant alertmanager. I noticed that the password for the smtp user is there in plain text. Howerver when I go the the multitenant alertmanager url I see that there the secret is blurred. -- auth_password: --

If I check in grafana with a singletenant alertmanager (https://github.com/grafana/prometheus-alertmanager) I see that the secret is also hidden for this alertmanager.

If I change the alertmanager implementation to prometheus instead of mimir, the secret is also hidden.

To Reproduce

Steps to reproduce the behavior:

1. install the multitenant alertmanager via mimir distributed helm chart.
2. push some alertmanager config with mimirtool that contain smtp settings
3. add the alertmanager as a datasource in grafana with implementation Mimir
4. Go to alerts -- contanct points and open the smtp configuration and check for the password.

Expected behavior

I expect that sensitive data is hidden and not displayed in plain text.

Environment

• Infrastructure: Kubernetes
• Deployment tool: helm

Additional Context

[ToonTijtgat2]

This is a duplicate of https://github.com/grafana/grafana/issues/81805 since I don't know if this is for the mimir team or for the grafana team.

[pstibrany]

This is a duplicate of grafana/grafana#81805 since I don't know if this is for the mimir team or for the grafana team.

Thank you. Mimir APIs returns full configuration without any obfuscation of secrets. I don't think Mimir team can do anything here.

[ToonTijtgat2]

@pstibrany thanks for the quick response, is this something that can be discussed with the grafana team? since it's kind of a security issue I don't feel comfortable using it when all users of grafana can read the password.

[pstibrany]

Let's bring this to attention of @grafana/alerting-squad .

[pstibrany]

Or @grafana/mimir-ruler-and-alertmanager-maintainers, which should be similar set of people.

[ToonTijtgat2]

How do I do that?

[pstibrany]

How do I do that?

I tried to assign the issue to @grafana/mimir-ruler-and-alertmanager-maintainers but that didn't quite work. I believe mentioning the groups like I did above triggers email notification to the people. I will also post link to this issue on internal Slack channel of the team.

[ToonTijtgat2]

How do I do that?

I tried to assign the issue to @grafana/mimir-ruler-and-alertmanager-maintainers but that didn't quite work. I believe mentioning the groups like I did above triggers email notification to the people. I will also post link to this issue on internal Slack channel of the team.

Thanks you!

[pstibrany]

Discussed internally in https://raintank-corp.slack.com/archives/C028MCV4R7C/p1706887511003569 thread. No conclusion about what's the best action forward yet.

[ToonTijtgat2]

Dear Thanks for the information, hopefully there will be a conclusion soon.

Have a good weekend.

[grobinson-grafana]

The issue is that user configurations in Mimir are stored in plaintext, and so plaintext secrets are visible in mimirtool and the Grafana UI. Mimir does not obfuscate secrets when reading user configurations via the Mimir API.

Until we add support for secrets backends in Mimir (such as Vault), this is how it has to work as otherwise it would break updating user configurations (for example, updating a user configuration via the Grafana UI would replace all actual secrets with the text <secret>).

In Prometheus Alertmanager, the API obfuscate all secrets with <secret> so the plaintext is never returned to the Grafana UI. This works in Prometheus AM as unlike in Mimir, you cannot update the Alertmanager configuration via the API. If you want to make changes to a Prometheus Alertmanager configuration you have to edit the file on disk.

The Grafana Multi-org Alertmanager has its own secrets backend that redacts secrets, allowing configurations to be redacted in the API, but un-redacted when loading the configuration into the Alertmanager.

I hope that explains the issue better.

If you have any questions, please let me know!

[ToonTijtgat2]

Hi

If I understand correctly, the only secure way to add a mimir alertmanager is then by doing it with the prometheus implementation, and add /alertmanager at the end of the url.

By adding the alertmanager as a prometheus type, the secret is not in plain text.

I'm just wondering what the added value is of the mimir alertmanager type, since it seems to have the same functionality as the prometheus one but this time with hidden secrets.

Thanks for the clarification. Toon Tijtgat

[grobinson-grafana]

I'm just wondering what the added value is of the mimir alertmanager type, since it seems to have the same functionality as the prometheus one but this time with hidden secrets.

With the Mimir implementation, the Grafana UI can be used to both read and write your Alertmanager configuration. For example, add/remove contact points, notification policies, etc.

With the Prometheus implementation you cannot do this as the configuration is shown as read-only in the Grafana UI. You can read your Alertmanager configuration, but you cannot make changes to it. If you want to make changes to it you will either need to use the Mimir implementation or use something like mimirtool.