stakater / Reloader

A Kubernetes controller to watch changes in ConfigMap and Secrets and do rolling upgrades on Pods with their associated Deployment, StatefulSet, DaemonSet and DeploymentConfig – [✩Star] if you're using it!
https://docs.stakater.com/reloader/
Apache License 2.0
7.45k stars 493 forks source link

Reloader not working properly with Strimzi Kafka Statefulsets #428

Open SilvM opened 1 year ago

SilvM commented 1 year ago

I managed to reproduce it in 2 different clusters of Openshift 4.9 and 4.10. I tried with reloader.stakater.com/auto: true and afterwards I tried with secret.reloader.stakater.com/reload: event-kafka-cert set at the spec.template.statefulset.metadata.annotations level.

I went ahead and modified the secret by adding a new field in data, both of which are detected right away

reloader-reloader-6475cfdc4b-wgqhd reloader-reloader time="2023-03-31T10:49:48Z" level=info msg="Changes detected in 'event-kafka-cert' of type 'SECRET' in namespace 'test-namespace', Updated 'event-kafka-kafka' of type 'StatefulSet' in namespace 'test-namespace'"
reloader-reloader-6475cfdc4b-wgqhd reloader-reloader time="2023-03-31T10:59:29Z" level=info msg="Changes detected in 'event-kafka-cert' of type 'SECRET' in namespace 'test-namespace', Updated 'event-kafka-kafka' of type 'StatefulSet' in namespace 'test-namespace'"

The pods' uptime however is unchanged at 50+days.

How can I troubleshoot this further?

hari819 commented 1 year ago

Strimzi has inbuilt mechanism in its operator to restart kafka brokers when the listener certificate gets renewed/changed,

A snippet from the strimzi doc is added below ,

Certificate changes or renewals If you need to change the signed certificate — for example, to add more DNS names or change some other settings — don’t worry. You can just update the Certificate resource and cert-manager will get a new updated certificate for you. Strimzi will automatically detect it and do a rolling update of the Kafka brokers to load the new certificate. Also certificate renewals will be done automatically by cert-manager and Strimzi.

https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/

SilvM commented 1 year ago

thanks for taking a look. I don't see any logs about that strimzi auto restart either