Add an interface allowing sending Alertmanager's configuration through the relation

[Gmerold]

Enhancement Proposal

Hello,

On behalf of the Telco team, I'd like to propose adding an interface similar to Prometheus's prometheus_remote_write, which would allow configuring the Alertmanager through the relation data. Main reason for this change is the fact, that Magma uses an external component, called alertmanager-configurer, to update Alertmanager with the configuration created via Magma NMS (GUI). Additionally, we believe that such change would add to the overall flexibility of the alertmanager-k8s charm.

From the implementation perspective, this would work just like in the Prometheus - consumer of the remote-write relation would push the config to the relation data bag and the charm would simply save this config to the alertmanager.yml file and reload the service.

Thanks in advance for looking into this and let the discussion begin.

BR, Bartek

[sed-i]

Thanks for the proposal, @Gmerold!

In the alertmanager charm we have intentionally avoided modeling the configuration file in the charm, and users are currently expected to provide the entire alertmanager configuration using two config option: config_file and templates_file.

Taking that information from relation data instead of a config option should not be too difficult, and would be somewhat analogous to the "config-builder" charms we already have, such as scrape-config and scrape-target. The proposed new relation could be another supported method for building alertmanager's configuration.

Some things to consider:

• config option takes precedence over relation data? (example: prom/231: external url config option takes precedence over relation data's ingress.url)
• would need to accommodate/reevaluate how alertmanager currently treats templates
• this would probably mean that the new relation interface would also override the tls relation alertmanager may already have (tls is not yet implemented for alertmanager, but is planned).
• currently I do not see any overlap between cli args and config file, but it could happen, and in the past there was an experimental arg that did just that: -web.config.file="" [EXPERIMENTAL] Path to configuration file that can enable TLS or authentication.. Need to think what would be the implications if a cli arg we're already using becomes available as a config file entry as well.

WDYT?

[Gmerold]

Thanks for replying Leon. In general, I agree with all your points.

As for the considerations you've mentioned:

• I'm completely fine with config option being the privileged one or, as we discussed in the call, we could change charm's status to Blocked and let the operator resolve the conflict.
• Regarding TLS, after taking another look at the Alertmanager Configurer's API, I realized that I was wrong saying it generates the whole config every time. Actually it allows you to overwrite only specific parts of the config (routes, receivers, global or templates). That would mean new relation needs to pass the current config from the Alermanager. But it also means, that potential TLS config doesn't necessarily have to be overridden.

[sed-i]

@balbirthomas @rbarry82 thoughts?

[balbirthomas]

Another special case may be worth considering. If Alertmanager exposes a relation interface that allows it to accept its entire configuration file and a set of notification templates is there also a need for conflict resolution or alternatively is only one such relation allowed ? I mean if two client charms are related to Alertmanager using this configuration relation interface that provides a config files, how should Alertmanger handle conflicts in two different configuration files ? More generally how should Alertmanager merge multiple configuration files (and notification templates) ? Note Prometheus charm does not support obtaining entire configuration file over relation data but only a very small subset i.e. "Static Scrape Jobs" and for this subset Prometheus ensures that Job names do not conflict.

[lucabello]

I think this is not relevant anymore, so closing; please feel free to reopen it though!