search

canonical / alertmanager-k8s-operator

This charmed operator automates operation procedures of Alertmanager, the alerting component of Prometheus and Loki, among others.
https://charmhub.io/alertmanager-k8s
Apache License 2.0
4 stars 16 forks source link

`certificates-relation-changed` hook failed: creating secrets: Regular expression is invalid: nothing to repeat #225

Open motjuste opened 4 months ago

motjuste commented 4 months ago

Solutions QA has observed this issue in one of our runs when deploying COS-lite on microk8s 1.28/stable where the non-primary alertmanager unit stays in error state.

The unit's debug logs in the crashdump show the following error being repeated:

unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "juju-log" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 WARNING unit.alertmanager/1.juju-log certificates:30: cannot determine if reload succeeded
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "status-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "juju-log" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG unit.alertmanager/1.juju-log certificates:30: Emitting custom event <CertChanged via AlertmanagerCharm/CertHandler[am-server-cert]/on/cert_changed[270]>.
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
unit-alertmanager-1: 2024-02-17 00:16:54 DEBUG jujuc running hook tool "relation-set" for alertmanager/1-certificates-relation-changed-172755738486250708
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.muxhttpserver received tls client hello for server name modeloperator.cos.svc
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=Role in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=RoleBinding in namespace cos
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.kubernetes.provider opening model "cos".
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.muxhttpserver received tls client hello for server name modeloperator.cos.svc
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for cn7vlr7mp25c7b4j91jg-1 of /v1, Kind=Secret in namespace cos
unit-alertmanager-1: 2024-02-17 00:16:55 ERROR juju.worker.uniter.context cannot apply changes: creating secrets: Regular expression is invalid: nothing to repeat
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of /v1, Kind=ServiceAccount in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=Role in namespace cos
model-a0121c38-4204-408f-82f9-6b08d0a522c3: 2024-02-17 00:16:55 DEBUG juju.worker.caasadmission received admission request for unit-alertmanager-1 of rbac.authorization.k8s.io/v1, Kind=RoleBinding in namespace cos
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.kubernetes.provider opening model "cos".
unit-alertmanager-1: 2024-02-17 00:16:55 ERROR juju.worker.uniter.operation hook "certificates-relation-changed" (via hook dispatching script: dispatch) failed: creating secrets: Regular expression is invalid: nothing to repeat
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.machinelock machine lock "machine-lock" released for alertmanager/1 uniter (run relation-changed (30; app: ca) hook)
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.worker.uniter.operation lock released for alertmanager/1
unit-alertmanager-1: 2024-02-17 00:16:55 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-alertmanager-1: 2024-02-17 00:16:55 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "certificates-relation-changed"
mmkay commented 2 months ago

Hi! Could you try with latest/stable (rev106) to confirm this is still an issue? There were recent changes in certificates-related code in alertmanager.