vmalert: adding `spec.notifier.url` creates `-notifier.url` with two URLs

[arseny-zinchenko]

Describe the bug

After adding a spec.notifier.url="http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager" to specify a new Alertmanager's URL, VMAlert has -notifier.url=http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093,http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager

This results to a lot of messages:

failed to send alerts to addr "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/api/v2/alerts": invalid SC 404 from "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/api/v2/alerts"; response body: 404 page not found

To Reproduce

Deploy VMAuth from the victoria-metrics-auth Helm chart v ~0.3.3.

Add to its values.yaml a new URL:

  vmalert:
    annotations: {}
    enabled: true
    spec:  
      notifier:
        url: "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager"

Deploy, and check corresponding Pod:

$ kk -n dev-monitoring-ns describe pod vmalert-vm-k8s-stack-554b89cdc5-96d59
...
Containers:
  vmalert:
    Container ID:  containerd://27a95f37d49d3dce175f3bba661d88779d3c7b813560405eeaf58a03bff15dda
    Image:         victoriametrics/vmalert:v1.93.0
    ...
    Args:
      -datasource.url=http://vmsingle-vm-k8s-stack.dev-monitoring-ns.svc:8429/
      -evaluationInterval=15s
      -httpListenAddr=:8080
      -notifier.url=http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093,http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager
      -remoteRead.url=http://vmsingle-vm-k8s-stack.dev-monitoring-ns.svc:8429/
...

Version

Image: victoriametrics/vmalert:v1.93.0

Logs

No response

Screenshots

Used command-line flags

No response

Additional information

No response

[Haleygo]

Hello! Got some mixed message here, can you clarify them?

Deploy VMAuth from the victoria-metrics-auth Helm chart v ~0.3.3.

You mean victoria-metrics-alert chart?

spec:  
      notifier:
        url: "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager"

There is no such field under victoria-metrics-alert, but notifier.alertmanager.url.

[setevoy2]

Hello! Got some mixed message here, can you clarify them?

Deploy VMAuth from the victoria-metrics-auth Helm chart v ~0.3.3.

You mean victoria-metrics-alert chart?

spec:  
      notifier:
        url: "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager"

There is no such field under victoria-metrics-alert, but notifier.alertmanager.url.

Sorry, I'm using victoria-metrics-k8s-stack. Have a lot of tabs opened, messed up a bit)

I found the notifier in the VMAlertSpec docs with a link to the VMAlertNotifierSpec, and link to it in the K8s Stack chart's values.

And it's working, as it added my "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager" But also left the old one.

And I've tried the notifier.alertmanager.url today, but it didn't work, and I went to the VMAlertSpec to find another way.

Yeah, just tested once again with:

  vmalert:
    annotations: {}
    enabled: true
    server:
      notifier: 
        alertmanager: 
          url: "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager" 

And now have -notifier.url=http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093, i.e. default. Am I using wrong formatting?

[Haleygo]

Seems like we have some hidden merge logic for vmalert spec here, can you create an issue to https://github.com/VictoriaMetrics/helm-charts since it's all about chart? And we will see how can we improve/clarify this)

[Haleygo]

If you are using victoria-metrics-k8s-stack chart, the vmalert.server you have is invalid since victoria-metrics-alert is not victoria-metrics-k8s-stack's dependency.

  vmalert:
    annotations: {}
    enabled: true
    server:
      notifier: 
        alertmanager: 
          url: "http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093/alertmanager" 

You should just use notifier,notifiers or notifierConfigRef under vmalert.spec. And the -notifier.url=http://vmalertmanager-vm-k8s-stack.dev-monitoring-ns.svc:9093 you saw is the default notifier from another template, it will be used when .Value.vmalert.spec.notifiers is empty, see merge logic here. https://github.com/VictoriaMetrics/helm-charts/blob/e36d247c1a69d08579d394d003101da1c4fda2f3/charts/victoria-metrics-k8s-stack/templates/_helpers.tpl#L156