nrvnrvn / k8dis

Redis Operator for Kubernetes
Apache License 2.0
5 stars 0 forks source link

Operator gets stuck at syncing the statefull set when container resources are specified in a non "normal" form #10

Open nrvnrvn opened 2 years ago

nrvnrvn commented 2 years ago

From https://github.com/amaizfinance/redis-operator/issues/45:

With this in my crd

 redis:
    image: redis:6.2.6-alpine
    initialDelaySeconds: 10
    resources:
      limits:
        cpu: 1000m
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 0.8Gi

the stateful set gets created with

        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 858993459200m

and the operator keeps logging ... "logger":"controller_redis","msg":"Applied *v1.StatefulSet","Namespace ... every 2 seconds

if I change my definition to

      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: 500m
        memory: 800Mi

the stateful set is created with identical values and the operator moves on to the next step.

P.S. To help other people find this issue: everything was "running" but there was no master label assigned to any of the pods, so I was unable to connect and the pod logs all showed the instances running as standalone servers, the ConfigMap did not have the replicaof line.

I guess something is normalising the resource values as the stateful set is created, but not normalising them before the diff check.


Original comment from @cmaster11:

Literally experienced the same issue today because we changed some resources fields.

What happened was that the new pods had no labels "role=master"/"role=replica", and the ConfigMap contained no "REPLICA OF" directive. @stocksr we found that our issue was related to using float numbers, meaning that we have been specifying limits and requests with e.g. "0.3". As soon as we switched to "300m" the issue went away.

Whenever we use any floats, we can see labels not appearing in the pods.