Flux worker thread inconsistency, creating 2 replica sets at the same time during first deployment

Describe the bug

I need some assistance. We are using the Gitops model (flux/helm) for auto deployment. We are observing an issue where 2 replica sets are getting created for a deployment at the same time. This is happening for the first deployment itself. This can cause potential inconsistencies.

Flux provides an option for configuring the worker count. On setting the worker count to 1, the above issue is not observed, which means that there is an inconsistency or race condition with the helm worker threads.

However, we cannot set the worker count to 1 for our production scenario since that will significantly slow down the setup. Has anyone faced this issue? Do you have any suggestions for a workaround?

Providing some details for clarity :

kubectl get rs -n test

NAME                          DESIRED    CURRENT    READY     AGE
mytest-service-84ddbfdd95        0          0         0       89s
mytest-service-ccc9dc478         1          1         1       89s

kubectl get deploy -n test

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
mytest-service          1/1       1             1        3m

Snippet from the describe deployment output -

Conditions:
  Type               Status  Reason
  ----               ------  ------
  Available           True    MinimumReplicasAvailable
  Progressing         True    NewReplicaSetAvailable

OldReplicaSets:  <none>
NewReplicaSet:   mytest-service-ccc9dc478 (1/1 replicas created)

Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  3m19s  deployment-controller  Scaled up replica set mytest-service-84ddbfdd95 to 1

  Normal  ScalingReplicaSet  3m19s  deployment-controller  Scaled up replica set mytest-service-ccc9dc478 to 1

  Normal  ScalingReplicaSet  2m39s  deployment-controller  Scaled down replica set mytest-service-84ddbfdd95 to 0

Snippet of the events log where we can see 2 replica sets getting created and scaled up at the same time. Shortly after, one of the replica sets gets scaled down to 0

kubectl get events -n test

LAST SEEN   TYPE      REASON              OBJECT                                                          MESSAGE
3m59s       Normal    Completed           job/mytest-service-jobs                             Job completed
3m59s       Normal    ScalingReplicaSet   deployment/mytest-service                           Scaled up replica set mytest-service-84ddbfdd95 to 1
3m59s       Normal    Scheduled           pod/mytest-service-84ddbfdd95-5ltbt                 Successfully assigned test/mytest-service-84ddbfdd95-5ltbt to master1.test-server
3m59s       Normal    SuccessfulCreate    replicaset/mytest-service-84ddbfdd95                Created pod: mytest-service-84ddbfdd95-5ltbt
3m59s       Normal    ScalingReplicaSet   deployment/mytest-service                           Scaled up replica set mytest-service-ccc9dc478 to 1
3m59s       Normal    SuccessfulCreate    replicaset/mytest-service-ccc9dc478                 Created pod: mytest-service-ccc9dc478-lkxdh
3m59s       Normal    Scheduled           pod/mytest-service-ccc9dc478-lkxdh                  Successfully assigned test/mytest-service-ccc9dc478-lkxdh to master1.test-server
3m58s       Normal    Created             pod/mytest-service-84ddbfdd95-5ltbt                 Created container mytest-service
3m57s       Normal    Started             pod/mytest-service-84ddbfdd95-5ltbt                 Started container mytest-service
3m19s       Normal    ScalingReplicaSet   deployment/mytest-service                           Scaled down replica set mytest-service-84ddbfdd95 to 0
3m19s       Normal    SuccessfulDelete    replicaset/mytest-service-84ddbfdd95                Deleted pod: mytest-service-84ddbfdd95-5ltbt
3m18s       Normal    Killing             pod/mytest-service-84ddbfdd95-5ltbt                 Stopping container mytest-service
3m16s       Normal    ReleaseSynced       helmrelease/mytest-service                          managed release 'mytest-service' in namespace 'test' synchronized

This is only the first deployment for the service, however I can see the deployment revision as 2

kubectl describe deploy mytest-service -n test

Name:                   mytest-service
CreationTimestamp:      Tue, 12 Apr 2022 09:43:35 +0000
Labels:                 app=mytest-service
                        app.kubernetes.io/managed-by=Helm
                        heritage=Helm

Annotations:            deployment.kubernetes.io/revision: 2
                        helm.fluxcd.io/antecedent: test:helmrelease/mytest-service
                        meta.helm.sh/release-name: mytest-service

I would greatly appreciate any help here!

Steps to reproduce

Install multiple applications with flux/helm

Expected behavior

During first deployment, one replica set should be created for an application

Kubernetes version / Distro / Cloud provider

K3S v1.21.1+k3s1

Flux version

flux v1.24.3 / Helm Chart 1.4.2 / fluxctl 1.24.3

Git provider

No response

Container Registry provider

No response

Additional context

No response

Maintenance Acknowledgement

[X] I am aware of Flux v1's maintenance status

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Creating a replicaset is not something that Flux itself does. Flux creates the Deployment and Kubernetes creates the replicaset to match its spec. When the spec changes, another replicaset gets created. It is surprising that two worker threads would start the same job at the same time without coordinating that between each other, (but I don't know the code well enough to say for sure that it does not happen like this...)

Unfortunately as you might have noticed, Flux v1 and Helm Operator are both approaching EOL, and all of the development is happening on Flux v2. This issue is not likely to be fixed in Flux v1, unless there is a patch which does not introduce any backwards incompatibility that is possible. Moreover, I am extremely constrained in the amount of effort that I can put into new development of features or bugfixes on Flux v1. We are supporting migrations away from Flux v1, and security patches if they are critical updates, but there is little help that I can offer with regard to fixing this issue if it is able to be confirmed.

The Migration Timetable that was published in the Flux docs marked Flux v1 and Helm Operator as superseded, in June 2021.

I cannot dig into the code right now to try to figure this issue out, but if you have a code fix or suggestion as to where the problem is, I can look into it further, and we could potentially merge a fix and release it if the fix can honor those constraints.

My primary suggestion though, is to begin migrating to Flux v2, where I suspect this issue will not be present anymore.

fluxcd / flux