When adding multiple Alertmanager configs in the plugin, messages are posted to a random channel

[aboukhal]

We're using Mattermost 9.4.1 and plugin 0.4.0

Setup: We have configured four different Alertmanagers in the cluster. They all use the same team, but different channels and they each have a unique token.

Incorrect behavior: When alerts are sent by the cluster (or when sending test messages) to the four URLs, the messages are being posted into one of the four configured channels randomly. When sending the same message over and over, the channel seems to be consistent, even if it is wrong. If the message is changed ever so slightly, it lands in a different, random channel, even though the token was not touched.

Expected behavior: The message should always be posted to the channel associated with the token.

What else have I tied: I looked at the code to determine if there were debug logs, that could be enabled to figure out what goes wrong. This is not the case. I also reviewed the relevant code sections but nothing jumped out immediately. It looks plausible.

What did I not try: I am unable to add additional debug logs or play with the code, as I'm not in an environment where I can easily build Go projects.

Do you have any idea what's going wrong?

[Pengwin61]

Good afternoon. I also encountered similar behavior of the plugin. I don't have an alertmanager cluster, only 1 copy. I created 3 channels in one team. And I get something completely different

route:
  receiver: 'telegram_bot'
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 1m
  repeat_interval: 12h
  routes:
  - match:
      alertgroup: 'SD'
    receiver: 'mm-home-sd'
  - match:
      alertgroup: 'InfraRules'
    receiver: 'mm-home-infra'
  - match:
      alertgroup: 'Hardware'
    receiver: 'mm-home-hardware'

[cpanato]

thanks for the report, are you willing to open a PR to fix this issue?

[aboukhal]

We don't have the environment to build Go applications without a lot of effort. So dry-analyzing the source code was as much as I could do at the moment

[aboukhal]

After figuring out that Mattermost supports most of the schema from Slack, we have now switched over to using the Slack configuration from Alertmanager. We've lost time calculations (using a timezone other than UTC or calculating durations, I've opened https://github.com/prometheus/alertmanager/issues/3762 for that) but gained extensive customizability. There are a lot of Slack template examples on the net to have a solid starting point for this. Thanks for this Plugin, we would have loved to stick with it but in the current state we needed a solution that we can easily develop ourselves.

[baznikin]

Confirm, same problem.

@aboukhal are you using Slack's configuation for Alertmanager to send notification to MatterMost, am I right?

[aboukhal]

@aboukhal are you using Slack's configuation for Alertmanager to send notification to MatterMost, am I right?

Correct. The tricky part is to create the templates so that messages look good in Mattermost with the appropriate formating, emojis, etc. But once that's done, it just works out of the box.