Support TLS certificate validation for receivers

[cbartz]

Enhancement Proposal

I am using the charm on edge rev 98 and am trying to use a slack receiver, but it fails with

:~$ kubectl logs alertmanager-0 -c alertmanager | less

2024-02-20T14:25:35.026Z [alertmanager] ts=2024-02-20T14:25:35.026Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
2024-02-20T14:25:35.027Z [alertmanager] ts=2024-02-20T14:25:35.027Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml
2024-02-20T14:25:35.181Z [alertmanager] ts=2024-02-20T14:25:35.181Z caller=notify.go:745 level=warn component=dispatcher receiver=mattermost-notifications integration=slack[0] aggrGroup="{}/{juju_model=\"prod-github-runner\"}:{juju_application=\"grafana-agent-without-dashboard\", juju_model=\"prod-github-runner\", juju_model_uuid=\"4572b1cc-0a39-40b7-818d-c68ed553f11a\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": x509: certificate signed by unknown authority"

The configuration I provided looks like this

global:
  http_config:
    tls_config:
      insecure_skip_verify: false
receivers:
- name: 'default'
- name: 'mattermost-notifications'
  slack_configs:
    - send_resolved: true
      api_url: 'https://fake-webhook-url'
route:
  group_by:
    - juju_application
    - juju_model_uuid
    - juju_model
  group_interval: 5m
  group_wait: 30s
  receiver: 'default'
  repeat_interval: 4h
  routes:
    - receiver: 'mattermost-notifications'
      matchers:
       - juju_model = prod-github-runner

The server in the api_url doesn't use a self-signed certificate, but one from Let's Encrypt. The reason certificate validation fails is probably because there are no ca certs in the container.

~$ kubectl exec -it alertmanager-0 -c alertmanager -- ls -l /etc/ssl/certs
total 0

I can fix this by setting insecure_skip_verify to true, but I think the charm should also support validation for common ca's or at least provide the ability to provide a ca file for https://prometheus.io/docs/alerting/latest/configuration/#tls_config.

[cbartz]

@Abuelodelanada /etc/ssl/certs/ is empty, this may be related to https://github.com/canonical/rockcraft/issues/343#issuecomment-1946506475

[Abuelodelanada]

Hi @cbartz !

Thanks for the report! We'll address this.

[cbartz]

This seems to have been fixed in the latest alertmanager (rev 107):

kubectl exec -it alertmanager-0  -c alertmanager -- ls -l /etc/ssl/certs                                                                                                                                                                                                     1 ↵
Defaulted container "charm" out of: charm, alertmanager, charm-init (init)
total 584

I'll try it out and will close this issue once confirmed.

[cbartz]

This has been fixed (tested with rev 107)