Incorrect web.external-url set

Bug Description

When pebble runs the alertmanager service it configures the flag --web.external-url which from what I can tell is,

The URL under which Alertmanager is externally reachable.

The alertmanager-k8s charm config for web-external-url states

Note: this config option shouldn't be included when you're using the "ingress" relation (e.g. traefik) - the charm will automatically assign an external url to --web.external-url when related to an ingress provider.

There seems to be a bug where the correct URL is not passed from the ingress relation through to pebble. This affects the URL that is present on alert messages. Instead of the public URL the local cluster URL is passed which makes it hard to view the alert.

To Reproduce

Note: I've deployed COS via a Terraform module and the issue is present, on two separate environments.

juju deploy alertmanger-k8s
juju deploy traefik-k8s
juju relate alertmanager-k8s traefik-k8s
Setup ingress address
kubectl exec -it alertmanager-k8s/0 -c alertmanager -- bash
/charm/bin/pebble plan

Observe incorrect address configured.

Environment

App           Version  Status  Scale  Charm             Channel        Rev  Address        Exposed  Message
alertmanager  0.27.0   active      1  alertmanager-k8s  latest/stable  125  10.87.175.146  no          
cos-ingress   2.11.0   active      1  traefik-k8s       latest/beta    203  10.87.50.99    yes      Serving at <my-address>

Relevant log output

Pebble Plan output:

# /charm/bin/pebble plan
services:
    alertmanager:
        summary: alertmanager service
        startup: enabled
        override: replace
        command: 'alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/alertmanager --web.listen-address=:9093 --cluster.listen-address= --web.external-url=http://alertmanager-0.alertmanager-endpoints.stg-cos-lite-comsys.svc.cluster.local:9093 '

Output from unit (only the relevant relations)

$ juju show-unit alertmanager/0
alertmanager/0:
  workload-version: 0.27.0
  opened-ports: []
  charm: ch:amd64/focal/alertmanager-k8s-125
  leader: true
  life: alive
  relation-info:
  - relation-id: 152
    endpoint: replicas
    related-endpoint: replicas
    application-data: {}
    local-unit:
      in-scope: true
      data:
        egress-subnets: 10.87.158.96/32
        ingress-address: 10.87.158.96
        private-address: 10.87.158.96
        private_address: http://alertmanager-0.alertmanager-endpoints.stg-cos-lite-comsys.svc.cluster.local:9093
  - relation-id: 156
    endpoint: catalogue
    related-endpoint: catalogue
    application-data: {}
    related-units:
      catalogue/0:
        in-scope: true
        data:
          egress-subnets: 10.87.48.243/32
          ingress-address: 10.87.48.243
          private-address: 10.87.48.243
  - relation-id: 158
    endpoint: ingress
    related-endpoint: ingress
    application-data:
      ingress: '{"url": "http://cos-ps6.staging.comsys.internal/stg-cos-lite-comsys-alertmanager"}'
    related-units:
      cos-ingress/0:
        in-scope: true
        data:
          egress-subnets: 10.87.115.180/32
          ingress-address: 10.87.115.180
          private-address: 10.87.115.180

canonical / alertmanager-k8s-operator