search

RedisLabs / redis-enterprise-k8s-docs

154 stars 91 forks source link

error while trying to run on openshift #89

Closed mrafieee closed 4 years ago

mrafieee commented 4 years ago

I am using : openshift 3.11 redis:5.4.6-18 operator:5.4.6-1186 k8s-controller:5.4.6-1186

We are trying to fetch from our own artifactory repo, this is the new feature enabled a week ago.

{"level":"error","ts":1573829086.3122225,"logger":"controller_redisenterprisecluster","msg":"Upgrade Error","Request.Namespace":"redis-cache-test","Request.Name":"cluster1-enterprise","error":"Upgrade Applied but will not start since REC Status.State is 'Upgrade' and not one of: 'Running', 'InvalidUpgrade'","stacktrace":"github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster.(*ReconcileRedisEnterpriseCluster).Reconcile\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster/redisenterprisecluster_controller.go:162\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

then it returns this: ` E1115 15:39:19.182190 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

`

mrafieee commented 4 years ago 
{"level":"info","ts":1573844130.9165604,"logger":"cmd","msg":"Go Version: go1.12.1"}
{"level":"info","ts":1573844130.9166677,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1573844130.916695,"logger":"cmd","msg":"Version of operator-sdk: v0.6.0"}
{"level":"info","ts":1573844130.9167016,"logger":"cmd","msg":"Service Rigger Version: 5.4.6-1186"}
{"level":"info","ts":1573844130.9167247,"logger":"cmd","msg":"Bootstrapper Version: 5.4.6-1186"}
{"level":"info","ts":1573844130.9171436,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1573844131.5281184,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."}
{"level":"info","ts":1573844131.528178,"logger":"leader","msg":"Continuing as the leader."}
{"level":"info","ts":1573844131.629587,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1573844131.6301644,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"redisenterprisecluster-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1573844131.727732,"logger":"cmd","msg":"failed to initialize service object for metrics: OPERATOR_NAME must be set"}
{"level":"info","ts":1573844131.7277975,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1573844131.8282204,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"redisenterprisecluster-controller"}
{"level":"info","ts":1573844131.9285111,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"redisenterprisecluster-controller","worker count":1}
{"level":"info","ts":1573844132.0293992,"logger":"controller_redisenterprisecluster","msg":"Verified RS upgrade not applied yet - checking upgrade is feasible","Request.Namespace":"redis-cache","Request.Name":"redis-enterprise"}
{"level":"error","ts":1573844132.0294604,"logger":"controller_redisenterprisecluster","msg":"Upgrade Error","Request.Namespace":"redis-cache","Request.Name":"redis-enterprise","error":"Upgrade Applied but will not start since REC Status.State is 'Upgrade' and not one of: 'Running', 'InvalidUpgrade'","stacktrace":"github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster.(*ReconcileRedisEnterpriseCluster).Reconcile\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster/redisenterprisecluster_controller.go:162\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1573844132.0295675,"logger":"controller_redisenterprisecluster","msg":"Error found during sync","error":"Upgrade Applied but will not start since REC Status.State is 'Upgrade' and not one of: 'Running', 'InvalidUpgrade'","stacktrace":"github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster.(*ReconcileRedisEnterpriseCluster).calcNewStatusForError\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster/redisenterprisecluster_controller.go:408\ngithub.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster.(*ReconcileRedisEnterpriseCluster).Reconcile\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster/redisenterprisecluster_controller.go:163\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1573844132.0457609,"logger":"controller_redisenterprisecluster","msg":"State changed from Upgrade to InvalidUpgrade."}
{"level":"info","ts":1573844132.0459838,"logger":"controller_redisenterprisecluster","msg":"Verified RS upgrade not applied yet - checking upgrade is feasible","Request.Namespace":"redis-cache","Request.Name":"redis-enterprise"}
{"level":"info","ts":1573844132.0460215,"logger":"controller_redisenterprisecluster","msg":"Upgrade validations completed successfully","Request.Namespace":"redis-cache","Request.Name":"redis-enterprise"}
{"level":"info","ts":1573844132.109968,"logger":"controller_redisenterprisecluster","msg":"State changed from InvalidUpgrade to Upgrade."}
E1115 18:55:32.582128       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/hashicorp/go-version/version.go:349
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/hashicorp/go-version/version.go:118
/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/redis_enterprise/resources.go:767
/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/redis_enterprise/resources.go:1157
/go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster/redisenterprisecluster_controller.go:157
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xf3e0a1]

goroutine 356 [running]:
github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x12f6a60, 0x234afd0)
    /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/hashicorp/go-version.(*Version).String(0x0, 0xc000b61f80, 0x147815e)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/hashicorp/go-version/version.go:349 +0x51
github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/hashicorp/go-version.(*Version).Compare(0x0, 0xc0001071d0, 0xc000131d40)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/github.com/hashicorp/go-version/version.go:118 +0x2f
github.com/RedisLabs/redis-enterprise-operator/pkg/redis_enterprise.NewStatefulSet(0xc00019af00, 0xc0000f37a0)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/redis_enterprise/resources.go:767 +0x1693
github.com/RedisLabs/redis-enterprise-operator/pkg/redis_enterprise.(*REResourcesHandler).DeployRedisEnterprise(0xc00052e3c0, 0x1689c40, 0xc0002afcb0, 0xc00019af00, 0x1, 0x0, 0x0)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/redis_enterprise/resources.go:1157 +0x451
github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster.(*ReconcileRedisEnterpriseCluster).Reconcile(0xc000123dc0, 0xc00076efc0, 0xb, 0xc00076efa0, 0x10, 0x235f260, 0x0, 0x0, 0x0)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/pkg/controller/redisenterprisecluster/redisenterprisecluster_controller.go:157 +0x80b
github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000e4be0, 0x0)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x1cc
github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1()
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36
github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000469830)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000469830, 0x3b9aca00, 0x0, 0x1, 0xc00078f560)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8
github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000469830, 0x3b9aca00, 0xc00078f560)
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
    /go/src/github.com/RedisLabs/redis-enterprise-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x311
`
danni-m commented 4 years ago

@mrafieee Hi! thanks for the bug report. Are you upgrading an existing cluster or this is a fresh install? Can you please upload the RedisEnterpriseCluster resource? (for example using kubectl: kubectl get rec -o yaml)

mrafieee commented 4 years ago

@danni-m here is how I build it using ansible:

apiVersion: "app.redislabs.com/v1alpha1"
kind: "RedisEnterpriseCluster"
metadata:
  name: "{{ item.name }}-enterprise"
  namespace: "{{ item.namespace }}"
spec:
  nodes: {{item.size}}
  pullSecrets:
     - name: {{docker.pull_secrets_id}}
  persistentSpec:
    enabled: true
    volumeSize: {{item.storage_size}}
    storageClassName: {{item.storage_class}}
  uiServiceType: LoadBalancer
  username: "{{ redis_cluster_admin_user }}"
  redisEnterpriseNodeResources:
    limits:
      cpu: "{{ item.cpu_cap }}"
      memory: "{{ item.memory_cap }}"
    requests:
      cpu: "1000m"
      memory: "1000Mi"
  redisEnterpriseImageSpec:
    imagePullPolicy:  Always
    repository:       {{ docker.registry}}/redis
    versionTag:       latest # 5.4.6-18
  redisEnterpriseServicesRiggerImageSpec:
    imagePullPolicy:  Always
    repository:       {{ docker.registry}}/k8s-controller
    versionTag:       latest # 5.4.6-1186
  bootstrapperImageSpec:
    imagePullPolicy:  Always
    repository:       {{ docker.registry}}/redis-operator
    versionTag:       latest # 5.4.6-1186

When I revert it back to use the redislabs dockerhub repo it started working fine . I built the images directly from the commented versions and pushed to our inhouse artifactory.

danni-m commented 4 years ago

@mrafieee you should use the actual version tags instead of latest. there are 2 reasons for that:

  1. this is the recommended approach for k8s, the reason for this is that latest can change and cause a issues with image caching on k8s nodes.
  2. we use the tag as a version indicator.
mrafieee commented 4 years ago

I tried using the same tags instead and it started working, just latest to the specific tag. thanks @danni-m