StatefulSet resources never created for RedisEnterpriseCluster

[grtrout]

After applying the bundle.yaml file, I applied my RedisEnterpriseCluster resource. Both succeeded.

kubectl apply -f bundle.yaml
role.rbac.authorization.k8s.io/redis-enterprise-operator created
rolebinding.rbac.authorization.k8s.io/redis-enterprise-operator created
secret/harbor-readonly-secret created
serviceaccount/redis-enterprise-operator created
customresourcedefinition.apiextensions.k8s.io/redisenterpriseclusters.app.redislabs.com created
deployment.apps/redis-enterprise-operator created
customresourcedefinition.apiextensions.k8s.io/redisenterprisedatabases.app.redislabs.com created

kubectl apply -f rec-test.yaml
redisenterprisecluster.app.redislabs.com/rec-test created

However, nothing happens after this. Usually I check the StatefulSet resources and wait for the associated pods to initialize. But, for some reason, that never happens.

kubectl get deployment -l name=redis-enterprise-operator
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
redis-enterprise-operator   1/1     1            1           30m

kubectl rollout status sts/rec-test
Error from server (NotFound): statefulsets.apps "rec-test" not found

Nothing seems to be failing and I see no clues in the namespace's events.

kubectl get events --sort-by=.metadata.creationTimestamp
LAST SEEN   TYPE     REASON              OBJECT                                           MESSAGE
32m         Normal   ScalingReplicaSet   deployment/redis-enterprise-operator             Scaled up replica set redis-enterprise-operator-64b4c9cd6 to 1
31m         Normal   Pulling             pod/redis-enterprise-operator-64b4c9cd6-tstss    Pulling image "harbor.company.com/project/redislabs/operator:6.0.12-5"
32m         Normal   SuccessfulCreate    replicaset/redis-enterprise-operator-64b4c9cd6   Created pod: redis-enterprise-operator-64b4c9cd6-tstss
31m         Normal   Scheduled           pod/redis-enterprise-operator-64b4c9cd6-tstss    Successfully assigned redis/redis-enterprise-operator-64b4c9cd6-tstss to cluster-workers-smzlj-6cc7b6df54-2t89l
31m         Normal   Pulled              pod/redis-enterprise-operator-64b4c9cd6-tstss    Successfully pulled image "harbor.company.com/project/redislabs/operator:6.0.12-5"
31m         Normal   Created             pod/redis-enterprise-operator-64b4c9cd6-tstss    Created container redis-enterprise-operator
31m         Normal   Started             pod/redis-enterprise-operator-64b4c9cd6-tstss    Started container redis-enterprise-operator

Any ideas? Things I should check? I've deployed the RedisEnterpriseCluster at least 5 times and never had this happen. Thank you.

[laurentdroin]

Hi,

The operator logs should show what the problem is: kubectl logs deploy/redis-enterprise-operator

I would suggest you open a Support Ticket with Redis Labs.

Laurent.

[grtrout]

Hey @laurentdroin While I can't say for certain why I didn't think to check the operator's logs, I appreciate the help! It was immediately obvious that there was a version mismatch.

{"level":"info","ts":"2021-04-21T20:14:00.635Z","logger":"controller_redisenterprisecluster","msg":"reconciliation will not continue until REC is upgraded","Request.Namespace":"redis","Request.Name":"rec-test","expected_version":"6.0.12-57"}

At some point I decided to use Redis version 6.0.12-58. I updated the RedisEnterpriseCluster definition to that version without thinking it could do any harm and since forgot.

Here's the part I'm referring to:

  redisEnterpriseImageSpec:
    imagePullPolicy: IfNotPresent
    repository: harbor.company.com/project/redislabs/redis
    versionTag: 6.0.12-58
  redisEnterpriseServicesRiggerImageSpec:
    imagePullPolicy: IfNotPresent
    repository: harbor.company.com/project/redislabs/k8s-controller
    versionTag: 6.0.12-5
  bootstrapperImageSpec:
    imagePullPolicy: IfNotPresent
    repository: harbor.company.com/project/redislabs/operator
    versionTag: 6.0.12-5

So, I changed the Redis version back to 6.0.12-57 and everything worked normally. Thanks again.