Reapplying KubeMod Manifests Breaks TLS

[jacobwoffenden]

Hey 👋

I'm currently working on an Ansible role to bootstrap a Kubernetes cluster, and I've come across an issue while reapplying the KubeMod manifest, it seems to break it's TLS.

I am applying a copy of https://raw.githubusercontent.com/kubemod/kubemod/v0.13.0/bundle.yaml and then deploying some rules, however if I reapply the bundle.yml and then my rules, I get the following

kubectl apply -f rules.yml 
Error from server (InternalError): error when applying patch:
{"spec":{"match":[{"matchValue":"Pod","select":"$.kind"},{"matchRegex":"zookeeper-.*","select":"$.metadata.name"}]}}
to:
Resource: "api.kubemod.io/v1beta1, Resource=modrules", GroupVersionKind: "api.kubemod.io/v1beta1, Kind=ModRule"
Name: "bau-zookeeper-node-affinity", Namespace: "bau"
for: "/opt/cba.homeoffice.gov.uk/kubernetes/kubemod-system/rules.yml": Internal error occurred: failed calling webhook "mmodrule.kubemod.io": Post "https://kubemod-webhook-service.kubemod-system.svc:443/mutate-api-kubemod-io-v1beta1-modrule?timeout=5s": x509: certificate signed by unknown authority

This is the output of reapplying bundle.yml

kubectl apply -f deploy.yml 
namespace/kubemod-system unchanged
customresourcedefinition.apiextensions.k8s.io/modrules.api.kubemod.io configured
role.rbac.authorization.k8s.io/kubemod-crt unchanged
clusterrole.rbac.authorization.k8s.io/kubemod-crt unchanged
clusterrole.rbac.authorization.k8s.io/kubemod-manager configured
rolebinding.rbac.authorization.k8s.io/kubemod-crt unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubemod-crt unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubemod-manager unchanged
service/kubemod-webapp-service unchanged
service/kubemod-webhook-service unchanged
deployment.apps/kubemod-operator unchanged
job.batch/kubemod-crt-job configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/kubemod-mutating-webhook-configuration configured
validatingwebhookconfiguration.admissionregistration.k8s.io/kubemod-validating-webhook-configuration configured

After a reapply, I cannot see any change in the kubemod-crt-job pod

kubectl -n kubemod-system logs kubemod-crt-job-v7t64
Generating certificates...
2021/08/02 23:23:36 [INFO] generating a new CA key and certificate from CSR
2021/08/02 23:23:36 [INFO] generate received request
2021/08/02 23:23:36 [INFO] received CSR
2021/08/02 23:23:36 [INFO] generating key: rsa-2048
2021/08/02 23:23:37 [INFO] encoded CSR
2021/08/02 23:23:37 [INFO] signed certificate with serial number 300404279375319863536081534241522847018381261362
2021/08/02 23:23:37 [INFO] generate received request
2021/08/02 23:23:37 [INFO] received CSR
2021/08/02 23:23:37 [INFO] generating key: rsa-2048
2021/08/02 23:23:37 [INFO] encoded CSR
2021/08/02 23:23:37 [INFO] signed certificate with serial number 628946871185316082866847940837242553210279511518
Generating Kubernetes certificate secret...
Applying certificate...
secret/webhook-server-cert created
Applying mutating webhook configuration patch...
mutatingwebhookconfiguration.admissionregistration.k8s.io/kubemod-mutating-webhook-configuration patched
Applying validating webhook configuration patch...
validatingwebhookconfiguration.admissionregistration.k8s.io/kubemod-validating-webhook-configuration patched

And finally, here are the logs from kubemod-operator pod

kubectl -n kubemod-system logs kubemod-operator-57d9c47ddf-26h4x
{"level":"info","ts":"2021-08-02 23:23:53.214Z","logger":"webapp-setup","msg":"web app server is starting to listen","addr":":8081"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8082"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"operator-setup","msg":"health server is starting to listen","addr":":8083"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.builder","msg":"Registering a mutating webhook","GVK":"api.kubemod.io/v1beta1, Kind=ModRule","path":"/mutate-api-kubemod-io-v1beta1-modrule"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-api-kubemod-io-v1beta1-modrule"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"api.kubemod.io/v1beta1, Kind=ModRule","path":"/validate-api-kubemod-io-v1beta1-modrule"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-api-kubemod-io-v1beta1-modrule"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"operator-setup","msg":"registering core mutating webhook"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.webhook","msg":"registering webhook","path":"/dragnet-webhook"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"operator-setup","msg":"starting manager"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":"2021-08-02 23:23:54.232Z","logger":"controller","msg":"Starting EventSource","reconcilerGroup":"api.kubemod.io","reconcilerKind":"ModRule","controller":"modrule","source":"kind source: /, Kind="}
{"level":"info","ts":"2021-08-02 23:23:54.233Z","logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":"2021-08-02 23:23:54.233Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2021-08-02 23:23:54.234Z","logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":"2021-08-02 23:23:54.234Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":"2021-08-02 23:23:54.333Z","logger":"controller","msg":"Starting Controller","reconcilerGroup":"api.kubemod.io","reconcilerKind":"ModRule","controller":"modrule"}
{"level":"info","ts":"2021-08-02 23:23:54.333Z","logger":"controller","msg":"Starting workers","reconcilerGroup":"api.kubemod.io","reconcilerKind":"ModRule","controller":"modrule","worker count":1}
2021/08/02 23:24:21 http: TLS handshake error from 10.244.2.1:46404: remote error: tls: bad certificate
...

To remedy this, I just delete the resources created by bundle.yml and reinstall.

I just wanted to check if I'm doing something wrong

---

Kubernetes Version 1.21.3

[vassilvk]

Hi @jacobwoffenden,

You need to delete kubemod-crt-job before re-applying (or upgrading) KubeMod’s manifests.

Please see instructions here: https://github.com/kubemod/kubemod#upgrade

This is necessary because re-deploying the KubeMod resources, such as web hooks configurations, wipes out the TLS certs, but the job that generates the certs won’t restart if it’s already there and is in “complete” state. Note that the job is the only resource you need to delete before re-applying KubeMod manifests.

[jacobwoffenden]

Cool, thanks @vassilvk! I'd ignored that because I wasn't upgrading KubeMod