Is there a way to only restart pods once, with both configmap and secrets change applied?

[ppb-ludekdolejsky]

We have noticed that when we change both configmap and secrets, first, new configmap is applied and pods restarted, and some time later, new secrets are applied and pods are restarted again.

So there is a period in between those two restart where pods are running with old secrets but new configmap, which is potentially a dangerous situation for us.

We are using "reloader.stakater.com/auto": "true"

[bnallapeta]

@ppb-ludekdolejsky This is an interesting situation indeed. Could you please elaborate on the "dangerous situation" you are ending up with these changes? Is it a case that the old secret and new configmap incompatibility will cause your application to crash? If that is the case, could you explain how Reloader can help here?

[ppb-ludekdolejsky]

@bnallapeta dangerous in a way that unless your healthcheck somehow catches this invalid combination and does not let the new version to roll out, you might end up with app that is running but not functional

Example: you switch to a different DB host using configmap, and update credentials using secrets

So no that the app crashes, but more like it rolls out successfully, but then does not work, causing a major incident

[bnallapeta]

@ppb-ludekdolejsky Ack. Do you have a solution in mind for this problem?

[ppb-ludekdolejsky]

we are going to test a workaround == improving our healthcheck to prevent such misconfigured pods from running, hoping that it will remove that interim (misconfigured) period (where only configmap got updated, but not yet secrets), but still allow the pods to restart once secrets get updated, too

it would be still nice if Reloader could restart the pods only once, after both configmap & secrets are updated - is it achievable?

[bnallapeta]

With the default implementation in place, k8s still creates two events - one for each change (configmap & secret). One possibility is to introduce a time delay before the restart so that both the configmap and secret are updated. But this comes with challenges:

• it is a very specific solution for your usecase.
• difficult to determine the time delay (30s, 1m, 2m etc)

We will discuss this within the team and see if this can be solved to serve a general usecase.

[bnallapeta]

@ppb-ludekdolejsky We discussed this and plan to implement the following:

• introduce a new annotation ' reloader.stakater.com/delay' to be set on the resources - deployment, statefulset, daemonset
• the annotation represents the time delay before the resource should reload. This way, the user can set an annotation to cause delay only the resources that they need.

[scartledge]

@bnallapeta Is there any ETA available for the proposed 'reloader.stakater.com/delay' annotation? Thanks.

[karl-johan-grahn]

@scartledge @ppb-ludekdolejsky Stakater does not have any internal use case for the enhancement request, so this work would be done by Stakater if you buy our Enterprise support. Otherwise, it will be a community effort to add it. You can email sales@stakater.com for our Enterprise support.

[kunj-bosamia]

I would like to work on this feature. Will come up with a some kind of brief approach and put it here in comments.