kube-oidc-proxy-orchestra pod died because the TLS secret is missing

[droslean]

  Warning  FailedMount  93s (x45 over 76m)  kubelet  MountVolume.SetUp failed for volume "kube-oidc-proxy-tls" : secret "unison-tls" not found

I am unsure if I remember correctly what is creating that secret, but I definitely don't understand how that secret got deleted or never existed.

[mlbiam]

The unison-tls Secret is created by the operator. It's the internal certificate used to secure communications between the ingress controller and OpenUnison&kube-oidc-proxy. If you "touch" the openunison orchestra object by adding an annotaion:

kubectl delete pods -l app=openunison-operator -n openunison
kubectl patch openunison orchestra -p '{"spec": {"template":{"metadata":{"annotations":{"force-update":"now"}}}} }'

The first command is so there's a clean set of logs for the operator in case the Secret doesn't get regenerated. The second command will trigger the operator to redeploy openunison, including generating any missing Secrets. If all goes well, both OpenUnison and kube-oidc-proxy should be all set.

[droslean]

@mlbiam spec.template.metadata.annotations on openunison resource doesn't exist.

Also the command generates error:

$ kubectl patch openunison orchestra -p '{"spec": {"template":{"metadata":{"annotations":{"force-update":"now"}}}} }'
Error from server (UnsupportedMediaType): the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json, application/apply-patch+yaml

[droslean]

@mlbiam I changed the metadata.annotations and it worked.

[mlbiam]

sorry, i copied&pasted too quick

[droslean]

How can we make sure that this won't happen again?

[mlbiam]

There's a known issue with the older versions of the cert-checker job that we fixed, so making sure you're on the latest charts is important. We're also simplifying the operator (and making it much smaller) to minimize issues like this happening in the future.