grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.15k stars 535 forks source link

mimirtool alertmanager verify: found character that cannot start any token #9409

Open toby181 opened 1 month ago

toby181 commented 1 month ago

Describe the bug

We're evaluating the mimirtool to verify / load the configuration and notification templates for the Alertmanager, using examples that are provided in the documentation. Doing so for the provided Alertmanager config works, doing the same for the provided notification policy fails with the exception "mimirtool: error: yaml: line 2: found character that cannot start any token, try --help".

Trying some other template will mostly result in "line 1: cannot unmarshal !!map into string".

To Reproduce

Steps to reproduce the behavior:

Verifying the Alertmanager configuration works, having this:

alertmanager_config.yaml:

route:
  receiver: "example_receiver"
  group_by: ["example_groupby"]
receivers:
  - name: "example_receiver"

Verification works (no exception)

$ mimirtool alertmanager verify alertmanager_config.yaml --log.level="debug"
INFO[0000] log level set to debug                       

Doing the same for the notification policy. alertmanager_template.tpl:

{{ define "alert_customer_env_message" }}
  [{{ .CommonLabels.alertname }} | {{ .CommonLabels.customer }} | {{ .CommonLabels.environment }}]
{{ end }}

Verification fails

$ mimirtool alertmanager verify alertmanager_template.tpl --log.level="debug"
INFO[0000] log level set to debug                       
mimirtool: error: yaml: line 2: found character that cannot start any token, try --help

Expected behavior

Either the documentation got a faulty example or there's some issue with the verification / linting.

Environment

Mimirtool is running in the latest version. Mimir itself is not involved yet.

mimirtool version  
Mimirtool, version 2.13.0 (branch: HEAD, revision: 4775ec156)
  go version:       go1.22.5
  platform:         darwin/arm64
checking latest version... You are on the latest version

Additional Context

armandgrillet commented 3 weeks ago

@tacole02 Can you have a look at this to verify that the docs are correct? If so we will need to then fix the code.

tacole02 commented 3 weeks ago

@armandgrillet , sure, I'll take a look!

tacole02 commented 2 weeks ago

Hi @toby181

@eamonryan kindly helped me to test this and find out what the issue was. If you look at the example under "Load AlertManager configuration" on this page, this is the input it expects:

mimirtool alertmanager verify <config_file> [template_files...]

with the .tpl file as optional input.

It looks like when you tested these configs, you didn't also include the alertmanager_config.yaml file before the template file. Doing so should work as expected.

Let me know if this helps!

derek-cadzow commented 6 days ago

@toby181 Can you confirm the solution outlined by @tacole02 works? We will close this item soon.

toby181 commented 4 days ago

Hi guys,

you're right @tacole02 , I did it without the alertmanager_config.yaml. Thanks! The command mimirtool alertmanager verify alertmanager_config.yaml alertmanager_template.tpl works.

But does it actually verify the alertmanager_template.tpl? For alertmanager_config.yaml it actually checks the syntax. Cause when I remove the {{ end }} it still passes.

{{ define "alert_customer_env_message" }}
  [{{ .CommonLabels.alertname }} | {{ .CommonLabels.customer }} | {{ .CommonLabels.environment }}]
tacole02 commented 4 days ago

Hi, Toby, I'm so glad the command worked!

I'm not sure about how the command itself is implemented and whether it verifies the syntax of the template file. @armandgrillet , do you know who we could check this with?

Thanks!

derek-cadzow commented 20 minutes ago

@armandgrillet, I believe this one is yours now and I'll remove it from Docs team (Grafana Labs wide)