Error on BottleRocket AMI: panic: runtime error: invalid memory address or nil pointer dereference

[VolodymyrSmahliuk]

What version of redis operator are you using?

kubectl logs redis-operator-54b9d96565-qtth8 - ot-operators

{"level":"info","ts":"2024-06-21T07:23:48Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-06-21T07:23:48Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-06-21T07:23:48Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-06-21T07:23:48Z","msg":"starting server","kind":"health probe","addr":"[::]:8081"}
I0621 07:23:48.835054       1 leaderelection.go:250] attempting to acquire leader lease ot-operators/6cab913b.redis.opstreelabs.in...
I0621 07:24:05.330521       1 leaderelection.go:260] successfully acquired lease ot-operators/6cab913b.redis.opstreelabs.in
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting EventSource","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","source":"kind source: *v1beta2.Redis"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting Controller","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting EventSource","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting Controller","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting EventSource","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster","source":"kind source: *v1beta2.RedisCluster"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting Controller","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting EventSource","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","source":"kind source: *v1beta2.RedisSentinel"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting EventSource","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","source":"kind source: *v1beta2.RedisReplication"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting Controller","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel"}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting workers","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","worker count":1}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting workers","controller":"rediscluster","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisCluster","worker count":1}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting workers","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","worker count":1}
{"level":"info","ts":"2024-06-21T07:24:05Z","msg":"Starting workers","controller":"redissentinel","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisSentinel","worker count":1}
{"level":"info","ts":"2024-06-21T07:24:05Z","logger":"controllers.Redis","msg":"Reconciling opstree redis controller","Request.Namespace":"thebigpos","Request.Name":"redis"}
{"level":"info","ts":"2024-06-21T07:24:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.storage.volumeClaimTemplate.metadata.creationTimestamp\""}
{"level":"error","ts":"2024-06-21T07:24:05Z","msg":"Reconciler error","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","Redis":{"name":"redis","namespace":"thebigpos"},"namespace":"thebigpos","name":"redis","reconcileID":"b5f6a5ad-9d23-4bd7-a97e-609e2d63d4d6","error":"Operation cannot be fulfilled on redis.redis.redis.opstreelabs.in \"redis\": StorageError: invalid object, Code: 4, Key: /registry/redis.redis.opstreelabs.in/redis/thebigpos/redis, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: ce15d080-7e78-4bc8-b913-34e5d949f960, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-06-21T07:24:05Z","logger":"controllers.Redis","msg":"Reconciling opstree redis controller","Request.Namespace":"thebigpos","Request.Name":"redis"}

redis-operator version: https://github.com/OT-CONTAINER-KIT/redis-operator/releases/tag/v0.17.0

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (kubectl version)?

OS: linux (amd64) OS Image: Bottlerocket OS 1.20.2 (aws-k8s-1.30) Kernel version: 6.1.90 Container runtime: containerd://1.6.31+bottlerocket Kubelet version: v1.30.0-eks-fff26e3

kubectl version Output

$ kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:27Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"30+", GitVersion:"v1.30.0-eks-036c24b", GitCommit:"59ddf7809432afedd41a880c1dfa8cedb39e5a1c", GitTreeState:"clean", BuildDate:"2024-04-30T23:53:46Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.27) and server (1.30) exceeds the supported minor version skew of +/-1

What did you do?

1. Followed the https://github.com/OT-CONTAINER-KIT/helm-charts/blob/main/charts/redis-operator/readme.md to install the redis-operator helm chart.
2. Followed the https://github.com/OT-CONTAINER-KIT/helm-charts/blob/main/charts/redis/README.md to install the stand-alone redis instance.

What did you expect to see?

The Stand-alone Redis Instance is up and running

What did you see instead?

Got an ERROR

```log {"level":"info","ts":"2024-06-21T06:59:34Z","logger":"controllers.Redis","msg":"Reconciling opstree redis controller","Request.Namespace":"thebigpos","Request.Name":"redis"} {"level":"info","ts":"2024-06-21T06:59:34Z","msg":"Starting workers","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","worker count":1} {"level":"info","ts":"2024-06-21T06:59:34Z", "msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference", "controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","Redis":{"name":"redis","namespace":"thebigpos"},"namespace":"thebigpos","name":"redis","reconcileID":"88f19a5e-dd19-41af-8db4-60d95e80b63b"} panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17a70e8] goroutine 144 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5 panic({0x19cf900?, 0x2cc0ca0?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getProbeInfo(0x0, 0x0?, 0x0, 0x0) /workspace/k8sutils/statefulset.go:617 +0x3e8 github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateContainerDef({_, _}, {{0xc00005b900, 0x1d}, {0xc0006229a0, 0xc}, 0x0, 0x0, {0xc000057680, 0x26}, ...}, ...) /workspace/k8sutils/statefulset.go:369 +0x159 github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateStatefulSetsDef({{0xc000622986, 0x5}, {0x0, 0x0}, {0xc000622990, 0x9}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...) /workspace/k8sutils/statefulset.go:234 +0x467 github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateOrUpdateStateFul({_, _}, {{_, _}, _}, {_, _}, {{0xc000622986, 0x5}, {0x0, ...}, ...}, ...) /workspace/k8sutils/statefulset.go:100 +0x1a5 github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateStandaloneRedis(0xc0005f38c0, {0x1f12bd0, 0xc000279860}) /workspace/k8sutils/redis-standalone.go:59 +0x853 github.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisReconciler).Reconcile(0xc000468690, {0x0?, 0x0?}, {{{0xc000622990?, 0x5?}, {0xc000622986?, 0xc0004d3d08?}}}) /workspace/controllers/redis_controller.go:67 +0x346 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1efc1d0?, {0x1ef8ed0?, 0xc000621b90?}, {{{0xc000622990?, 0xb?}, {0xc000622986?, 0x0?}}}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003019a0, {0x1ef8f08, 0xc0001c4af0}, {0x1a86860?, 0xc000051040?}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3cc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003019a0, {0x1ef8f08, 0xc0001c4af0}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1af sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 69 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x565 ```

Full log in the attachments: redis-operator.log

[VolodymyrSmahliuk]

I saw the similar issues, but the previous comment https://github.com/OT-CONTAINER-KIT/redis-operator/issues/921#issuecomment-2105510477 doesn't work for me

[jradikk]

Got a similar issue with Ubuntu 24, Kubernetes 1.28, and the latest operator version

[drivebyer]

@VolodymyrSmahliuk Which version of the Redis operator chart are you using?

[VolodymyrSmahliuk]

@drivebyer I use the latest version (0.16.4). Here is my helm install command for both redis-operator and redis:

helm upgrade -i redis-operator --namespace ot-operators ot-helm/redis-operator --version 0.16.4
helm upgrade -i redis --namespace ot-operators ot-helm/redis --version 0.16.0

[drivebyer]

Oh, I think it's already fixed at https://github.com/OT-CONTAINER-KIT/redis-operator/pull/934. @VolodymyrSmahliuk

[VolodymyrSmahliuk]

@drivebyer, great!

I see that the fix is in master branch, but no release yet. Is there any pre-release image that includes the fix?

[drivebyer]

@VolodymyrSmahliuk No, there is not. However, you can work around it by setting livenessProbe and readinessProbe in the custom resource, like this:

---
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: Redis
metadata:
  name: redis-standalone
spec:
  # redisConfig:
  #   additionalRedisConfig: redis-external-config
  podSecurityContext:
    runAsUser: 1000
    fsGroup: 1000
  readinessProbe:
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 3
  livenessProbe:
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 3
  kubernetesConfig:
    image: quay.io/opstree/redis:v7.0.12
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
        # redisSecret:
        #   name: redis-secret
        #   key: password
        # imagePullSecrets:
        #   - name: regcred
  redisExporter:
    enabled: false
    image: quay.io/opstree/redis-exporter:v1.44.0
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
        # Environment Variables for Redis Exporter
        # env:
        # - name: REDIS_EXPORTER_INCL_SYSTEM_METRICS
        #   value: "true"
        # - name: UI_PROPERTIES_FILE_NAME
        #   valueFrom:
        #     configMapKeyRef:
        #       name: game-demo
        #       key: ui_properties_file_name
        # - name: SECRET_USERNAME
        #   valueFrom:
        #     secretKeyRef:
        #       name: mysecret
        #       key: username
  storage:
    volumeClaimTemplate:
      spec:
        # storageClassName: standard
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
            # nodeSelector:
            #   kubernetes.io/hostname: minikube
            # podSecurityContext: {}
            # priorityClassName:
            # affinity:
            # Tolerations: []

[VolodymyrSmahliuk]

@drivebyer, thank you for quick reply!

I just run your manifest, and got an error when running kubectl apply:

Error from server (BadRequest): error when creating "redis.yaml": Redis in version "v1beta2" cannot be handled as a Redis: strict decoding error: unknown field "spec.livelinessProbe"

Which is obvious, because of typo in spec.livelinessProbe -> spec.livenessProbe

However, the redis pod started creating. Thank you for your help!

PS: I still can't run the redis pod, but it's a problem with PVC. Will fix that 👌

[drivebyer]

Sorry for the typo. Please change livelinessProbe to livenessProbe @VolodymyrSmahliuk.