search

kedacore / keda-olm-operator

Operator for deploying KEDA Controller on OperatorHub.io/OLM
Apache License 2.0
31 stars 23 forks source link

Keda external metrics endpoint breaks after OpenShift cert rotation #11

Closed bbrowning closed 4 years ago

bbrowning commented 4 years ago

Expected Behavior

Keda's external metrics apiservice endpoint should continue to work on a long-running OpenShift cluster.

Actual Behavior

Keda's external metrics apiservice endpoint stops working as soon as OpenShift rotates certificates.

Steps to Reproduce the Problem

  1. Install the KEDA operator
  2. oc get apiservice v1beta1.external.metrics.k8s.io and verify Available is True.
  3. Wait for OpenShift to rotate certificates - 24 hours after creation of a brand new cluster, periodically after that
  4. oc get apiservice v1beta1.external.metrics.k8s.io and see Available is now False.

Specifications

bbrowning commented 4 years ago 
$ oc get apiservice v1beta1.external.metrics.k8s.io
NAME                              SERVICE                       AVAILABLE                      AGE
v1beta1.external.metrics.k8s.io   keda/keda-metrics-apiserver   False (FailedDiscoveryCheck)   21d

$ oc logs -n keda deployment/keda-metrics-apiserver --tail 5
E0416 17:41:23.982673       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0416 17:41:23.982705       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0416 17:41:23.982785       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0416 17:41:23.982673       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
E0416 17:41:23.982891       1 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
bbrowning commented 4 years ago

If installing KEDA on an existing OpenShift cluster that's past the initial cert rotation at the 24 hour mark, rotation will happen at least once every 30 days. I'm not sure if there's a way to manually force rotation to happen on-demand.