search

OT-CONTAINER-KIT / redis-operator

A golang based redis operator that will make/oversee Redis standalone/cluster/replication/sentinel mode setup on top of the Kubernetes.
https://ot-redis-operator.netlify.app/
Apache License 2.0
829 stars 229 forks source link

invalid memory address or nil pointer dereference #132

Closed MohamadTahir closed 3 years ago

MohamadTahir commented 3 years ago

hey I've installed redis-operator using the helm chart but it keeps restarting, the pods gives me the below logs, I have 3 stand alones which are running fine.

I0816 10:18:18.814673 1 request.go:645] Throttling request took 1.047546143s, request: GET:https://10.43.0.1:443/apis/rbacmanager.reactiveops.io/v1beta1?timeout=32s

Mon, Aug 16 2021 1:18:19 pm | 2021-08-16T10:18:19.369Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"} Mon, Aug 16 2021 1:18:19 pm | 2021-08-16T10:18:19.371Z INFO setup starting manager Mon, Aug 16 2021 1:18:19 pm | I0816 10:18:19.371782 1 leaderelection.go:243] attempting to acquire leader lease redis-operator/6cab913b.redis.opstreelabs.in... Mon, Aug 16 2021 1:18:19 pm | 2021-08-16T10:18:19.371Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"} Mon, Aug 16 2021 1:18:36 pm | I0816 10:18:36.850622 1 leaderelection.go:253] successfully acquired lease redis-operator/6cab913b.redis.opstreelabs.in Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.850Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"redis-operator","name":"6cab913b.redis.opstreelabs.in","uid":"586e974a-6d7e-4ae9-9833-77cbb9c48e66","apiVersion":"v1","resourceVersion":"104861282"}, "reason": "LeaderElection", "message": "redis-operator-67ff7665db-cggfg_864e9485-d051-498c-b369-aa322f3993e7 became leader"} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.850Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Lease","namespace":"redis-operator","name":"6cab913b.redis.opstreelabs.in","uid":"923d6cd6-b41a-407f-a072-d84535bc5f26","apiVersion":"coordination.k8s.io/v1","resourceVersion":"104861283"}, "reason": "LeaderElection", "message": "redis-operator-67ff7665db-cggfg_864e9485-d051-498c-b369-aa322f3993e7 became leader"} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.850Z INFO controller-runtime.manager.controller.rediscluster Starting EventSource {"reconciler group": "redis.redis.opstreelabs.in", "reconciler kind": "RedisCluster", "source": "kind source: /, Kind="} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.850Z INFO controller-runtime.manager.controller.redis Starting EventSource {"reconciler group": "redis.redis.opstreelabs.in", "reconciler kind": "Redis", "source": "kind source: /, Kind="} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.951Z INFO controller-runtime.manager.controller.rediscluster Starting Controller {"reconciler group": "redis.redis.opstreelabs.in", "reconciler kind": "RedisCluster"} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.951Z INFO controller-runtime.manager.controller.rediscluster Starting workers {"reconciler group": "redis.redis.opstreelabs.in", "reconciler kind": "RedisCluster", "worker count": 1} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.951Z INFO controller-runtime.manager.controller.redis Starting Controller {"reconciler group": "redis.redis.opstreelabs.in", "reconciler kind": "Redis"} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.951Z INFO controller-runtime.manager.controller.redis Starting workers {"reconciler group": "redis.redis.opstreelabs.in", "reconciler kind": "Redis", "worker count": 1} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.952Z INFO controllers.Redis Reconciling opstree redis controller {"Request.Namespace": "redis-standalone", "Request.Name": "standalone-1"} Mon, Aug 16 2021 1:18:36 pm | 2021-08-16T10:18:36.957Z INFO controller_redis Redis statefulset get action failed {"Request.StateFulSet.Namespace": "redis-standalone", "Request.StateFulSet.Name": "standalone-1"} Mon, Aug 16 2021 1:18:36 pm | E0816 10:18:36.957495 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) Mon, Aug 16 2021 1:18:36 pm | goroutine 288 [running]: Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1672660, 0x23e10c0) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/runtime/runtime.go:74 +0xa6 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/runtime/runtime.go:48 +0x89 Mon, Aug 16 2021 1:18:36 pm | panic(0x1672660, 0x23e10c0) Mon, Aug 16 2021 1:18:36 pm | /usr/local/go/src/runtime/panic.go:969 +0x1b9 Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.generateContainerDef(0xc0005276a0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000142cc0, 0x22, 0xc000527746, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/statefulset.go:153 +0x33d Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.generateStateFulSetsDef(0xc0005276a0, 0xc, 0x0, 0x0, 0xc0005276c0, 0x10, 0x0, 0x0, 0x0, 0x0, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/statefulset.go:107 +0xc7 Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.CreateOrUpdateStateFul(0xc0005276c0, 0x10, 0xc0005276a0, 0xc, 0x0, 0x0, 0xc0005276c0, 0x10, 0x0, 0x0, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/statefulset.go:53 +0x2b8 Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.CreateStandAloneRedis(0xc0002ba240, 0xc0002ba240, 0x1a52220) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/redis-standalone.go:40 +0x7f8 Mon, Aug 16 2021 1:18:36 pm | redis-operator/controllers.(RedisReconciler).Reconcile(0xc00070a0f0, 0x1a2db60, 0xc0002722d0, 0xc0005276c0, 0x10, 0xc0005276a0, 0xc, 0xc0002722d0, 0x40a27f, 0xc000030000, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/controllers/redis_controller.go:59 +0x328 Mon, Aug 16 2021 1:18:36 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc000314000, 0x1a2daa0, 0xc0001ae000, 0x16c9b80, 0xc000436b20) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263 +0x317 Mon, Aug 16 2021 1:18:36 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc000314000, 0x1a2daa0, 0xc0001ae000, 0xc0002f2e00) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235 +0x205 Mon, Aug 16 2021 1:18:36 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1.1(0x1a2daa0, 0xc0001ae000) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198 +0x4a Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1() Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0x37 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0002f2f50) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155 +0x5f Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000411f50, 0x19f7340, 0xc000272150, 0xc0001ae001, 0xc00056a000) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156 +0xad Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002f2f50, 0x3b9aca00, 0x0, 0x1, 0xc00056a000) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133 +0x98 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1a2daa0, 0xc0001ae000, 0xc00013e180, 0x3b9aca00, 0x0, 0x1) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0xa6 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1a2daa0, 0xc0001ae000, 0xc00013e180, 0x3b9aca00) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99 +0x57 Mon, Aug 16 2021 1:18:36 pm | created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1 Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:195 +0x4e7 Mon, Aug 16 2021 1:18:36 pm | panic: runtime error: invalid memory address or nil pointer dereference [recovered] Mon, Aug 16 2021 1:18:36 pm | panic: runtime error: invalid memory address or nil pointer dereference Mon, Aug 16 2021 1:18:36 pm | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x15366fd] Mon, Aug 16 2021 1:18:36 pm |   Mon, Aug 16 2021 1:18:36 pm | goroutine 288 [running]: Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/runtime/runtime.go:55 +0x10c Mon, Aug 16 2021 1:18:36 pm | panic(0x1672660, 0x23e10c0) Mon, Aug 16 2021 1:18:36 pm | /usr/local/go/src/runtime/panic.go:969 +0x1b9 Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.generateContainerDef(0xc0005276a0, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc000142cc0, 0x22, 0xc000527746, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/statefulset.go:153 +0x33d Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.generateStateFulSetsDef(0xc0005276a0, 0xc, 0x0, 0x0, 0xc0005276c0, 0x10, 0x0, 0x0, 0x0, 0x0, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/statefulset.go:107 +0xc7 Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.CreateOrUpdateStateFul(0xc0005276c0, 0x10, 0xc0005276a0, 0xc, 0x0, 0x0, 0xc0005276c0, 0x10, 0x0, 0x0, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/statefulset.go:53 +0x2b8 Mon, Aug 16 2021 1:18:36 pm | redis-operator/k8sutils.CreateStandAloneRedis(0xc0002ba240, 0xc0002ba240, 0x1a52220) Mon, Aug 16 2021 1:18:36 pm | /workspace/k8sutils/redis-standalone.go:40 +0x7f8 Mon, Aug 16 2021 1:18:36 pm | redis-operator/controllers.(RedisReconciler).Reconcile(0xc00070a0f0, 0x1a2db60, 0xc0002722d0, 0xc0005276c0, 0x10, 0xc0005276a0, 0xc, 0xc0002722d0, 0x40a27f, 0xc000030000, ...) Mon, Aug 16 2021 1:18:36 pm | /workspace/controllers/redis_controller.go:59 +0x328 Mon, Aug 16 2021 1:18:36 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc000314000, 0x1a2daa0, 0xc0001ae000, 0x16c9b80, 0xc000436b20) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263 +0x317 Mon, Aug 16 2021 1:18:36 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc000314000, 0x1a2daa0, 0xc0001ae000, 0xc0002f2e00) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235 +0x205 Mon, Aug 16 2021 1:18:36 pm | sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1.1(0x1a2daa0, 0xc0001ae000) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198 +0x4a Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1() Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0x37 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0002f2f50) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155 +0x5f Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000411f50, 0x19f7340, 0xc000272150, 0xc0001ae001, 0xc00056a000) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156 +0xad Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002f2f50, 0x3b9aca00, 0x0, 0x1, 0xc00056a000) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133 +0x98 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1a2daa0, 0xc0001ae000, 0xc00013e180, 0x3b9aca00, 0x0, 0x1) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:185 +0xa6 Mon, Aug 16 2021 1:18:36 pm | k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1a2daa0, 0xc0001ae000, 0xc00013e180, 0x3b9aca00) Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:99 +0x57 Mon, Aug 16 2021 1:18:36 pm | created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func1 Mon, Aug 16 2021 1:18:36 pm | /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:195 +0x4e7

iamabhishek-dubey commented 3 years ago

What is the version of the redis operator?

MohamadTahir commented 3 years ago

the latest update version on helm.

iamabhishek-dubey commented 3 years ago

Execute helm repo update and install redis again after removing it.

MohamadTahir commented 3 years ago

Same thing.

iamabhishek-dubey commented 3 years ago

It looks like a version mismatch of your operator and definition of redis. Have you upgraded the redis-operator as well.

here are the things

you can check this by

helm ls --namespace <namespace>
MohamadTahir commented 3 years ago

the redis operator version is 0.7.0 but we are not using helm for installing the stand alone because of the nil error that was occurring in the pervious version

this is how we create our redis stand alone

apiVersion: redis.redis.opstreelabs.in/v1beta1
kind: Redis
metadata:
  name: standalone-INSTANCE
  namespace: redis-standalone
spec:
  mode: standalone
  global:
    image: quay.io/opstree/redis:v6.2
    imagePullPolicy: IfNotPresent
    existingPasswordSecret:
      name: redis-secret
      key: password
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: "2"
        memory: 2Gi
  service:
    type: ClusterIP
  redisConfig: {}
  redisExporter:
    enabled: true
    image: quay.io/opstree/redis-exporter:1.0
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
iamabhishek-dubey commented 3 years ago

You are missing this https://github.com/OT-CONTAINER-KIT/redis-operator/blob/master/example/redis-standalone.yaml#L42

iamabhishek-dubey commented 3 years ago

this is something revamped

MohamadTahir commented 3 years ago

yes but the stand alones are working fine without any problem? previously everything was working correctly but now it started failing.

iamabhishek-dubey commented 3 years ago

Yes because you have upgraded the redis operator and it has breaking changes. Either update the CRD, or downgrade redis operator to the older version

iamabhishek-dubey commented 3 years ago

Closing this issue because it's not an actual bug/feature