search

kudobuilder / kudo

Kubernetes Universal Declarative Operator (KUDO)
https://kudo.dev
Apache License 2.0
1.18k stars 103 forks source link

KUDO crashes on null parameter values #1602

Closed zmalik closed 4 years ago

zmalik commented 4 years ago

What happened: Using a KUDOOperator Task deploying through a params file, using a not required param crashloop the KUDO manager from which it cannot recover

.
└── operator
    ├── operator.yaml
    ├── params.yaml
    └── templates
        └── zk-params.yaml

./operator/operator.yaml:

apiVersion: kudo.dev/v1beta1
name: "first-operator"
operatorVersion: "0.2.0"
kubernetesVersion: 1.13.0
maintainers:
  - name: Your name
    email: <your@email.com>
url: https://kudo.dev
tasks:
  - name: deploy-zookeeper
    kind: KudoOperator
    spec:
      package: zookeeper
      operatorVersion: 0.3.0
      parameterFile: zk-params.yaml
      instanceName: zookeeper-instance
plans:
  deploy:
    strategy: serial
    phases:
      - name: main
        strategy: parallel
        steps:
          - name: everything
            tasks:
              - deploy-zookeeper

./operator/params.yaml:

apiVersion: kudo.dev/v1beta1
parameters:
  - name: ZK_STORAGE_CLASS
    required: false

./operator/templates/zk-params.yaml:

STORAGE_CLASS: {{ .Params.ZK_STORAGE_CLASS }}

This crashloops the kudo manager

kubectl get pods -n kudo-system
NAME                        READY   STATUS             RESTARTS   AGE
kudo-controller-manager-0   0/1     CrashLoopBackOff   5          4m37s

with error:

2020/07/14 17:04:15 InstanceController: Going to proceed with execution of the scheduled plan 'deploy' on instance default/first-operator-instance
E0714 17:04:15.835303       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 396 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x155b740, 0x247ff80)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/runtime/runtime.go:48 +0x82
panic(0x155b740, 0x247ff80)
    /usr/local/Cellar/go/1.14.4/libexec/src/runtime/panic.go:969 +0x166
github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params.GetParametersFromFile(0xc00082ed70, 0xe, 0xc000520230, 0xf, 0x10, 0xc0007f29c0, 0xc0004de8f0, 0x7)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params/parser.go:93 +0x485
github.com/kudobuilder/kudo/pkg/engine/task.instanceParameters(0xc00082ed70, 0xe, 0xc0007f21e0, 0xc0005c67c0, 0x17, 0xc0004de8f0, 0x7, 0xc00082eca0, 0xe, 0xc000813ba0, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/engine/task/task_kudo_operator.go:101 +0x28f
github.com/kudobuilder/kudo/pkg/engine/task.KudoOperatorTask.Run(0xc00082ed10, 0x10, 0xc00082ed50, 0x9, 0xc000813bc0, 0x12, 0x0, 0x0, 0xc00082ed40, 0x5, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/engine/task/task_kudo_operator.go:48 +0x1f9
github.com/kudobuilder/kudo/pkg/engine/workflow.Execute(0xc0003ae230, 0xc0008bda70, 0x1976640, 0xc0005d1ad0, 0x197d0c0, 0xc0002c0400, 0xc0000f4400, 0xc000197f80, 0x20, 0x1495700, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/engine/workflow/engine.go:181 +0x10ab
github.com/kudobuilder/kudo/pkg/controller/instance.(*Reconciler).Reconcile(0xc0002c0480, 0xc0004de8f0, 0x7, 0xc0005c67c0, 0x17, 0x0, 0xbfbb9743f157ba44, 0xc00046e630, 0xc000146638)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/controller/instance/instance_controller.go:235 +0xa54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006ba300, 0x15b34a0, 0xc000524040, 0x0)
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256 +0x161
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006ba300, 0x203000)
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232 +0xae
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0006ba300)
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000440400)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000440400, 0x192c340, 0xc0007f2120, 0x1, 0xc000447200)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000440400, 0x3b9aca00, 0x0, 0xc0000c6b01, 0xc000447200)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000440400, 0x3b9aca00, 0xc000447200)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:193 +0x305
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13f57b5]

goroutine 396 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/runtime/runtime.go:55 +0x105
panic(0x155b740, 0x247ff80)
    /usr/local/Cellar/go/1.14.4/libexec/src/runtime/panic.go:969 +0x166
github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params.GetParametersFromFile(0xc00082ed70, 0xe, 0xc000520230, 0xf, 0x10, 0xc0007f29c0, 0xc0004de8f0, 0x7)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params/parser.go:93 +0x485
github.com/kudobuilder/kudo/pkg/engine/task.instanceParameters(0xc00082ed70, 0xe, 0xc0007f21e0, 0xc0005c67c0, 0x17, 0xc0004de8f0, 0x7, 0xc00082eca0, 0xe, 0xc000813ba0, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/engine/task/task_kudo_operator.go:101 +0x28f
github.com/kudobuilder/kudo/pkg/engine/task.KudoOperatorTask.Run(0xc00082ed10, 0x10, 0xc00082ed50, 0x9, 0xc000813bc0, 0x12, 0x0, 0x0, 0xc00082ed40, 0x5, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/engine/task/task_kudo_operator.go:48 +0x1f9
github.com/kudobuilder/kudo/pkg/engine/workflow.Execute(0xc0003ae230, 0xc0008bda70, 0x1976640, 0xc0005d1ad0, 0x197d0c0, 0xc0002c0400, 0xc0000f4400, 0xc000197f80, 0x20, 0x1495700, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/engine/workflow/engine.go:181 +0x10ab
github.com/kudobuilder/kudo/pkg/controller/instance.(*Reconciler).Reconcile(0xc0002c0480, 0xc0004de8f0, 0x7, 0xc0005c67c0, 0x17, 0x0, 0xbfbb9743f157ba44, 0xc00046e630, 0xc000146638)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/controller/instance/instance_controller.go:235 +0xa54
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006ba300, 0x15b34a0, 0xc000524040, 0x0)
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256 +0x161
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006ba300, 0x203000)
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232 +0xae
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc0006ba300)
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000440400)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000440400, 0x192c340, 0xc0007f2120, 0x1, 0xc000447200)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000440400, 0x3b9aca00, 0x0, 0xc0000c6b01, 0xc000447200)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000440400, 0x3b9aca00, 0xc000447200)
    /Users/kensipe/projects/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /Users/kensipe/projects/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:193 +0x305

What you expected to happen:

using zk-params.yaml as STORAGE_CLASS: "{{ .Params.ZK_STORAGE_CLASS }}" fixes the issue

How to reproduce it (as minimally and precisely as possible): Above example

Environment:

porridge commented 4 years ago

Turns out this is not specific to the KUDOOperator task. It also happens on plain command line when a null parameter value is supplied.

$ cat a.yaml 
a:
b: 2
$ kubectl kudo install kafka -P a.yaml 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14050bc]

goroutine 1 [running]:
github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params.getParamsFromFiles(0x1c04940, 0x294e670, 0xc000327ba0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params/parser.go:79 +0x97c
github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params.GetParameterMap(0x1c04940, 0x294e670, 0x0, 0x0, 0x0, 0xc000327ba0, 0x1, 0x1, 0x4, 0x193fe61, ...)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/kudoctl/cmd/params/parser.go:23 +0xca
github.com/kudobuilder/kudo/pkg/kudoctl/cmd.newInstallCmd.func1(0xc0006b82c0, 0xc000686840, 0x1, 0x3, 0x0, 0x0)
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/pkg/kudoctl/cmd/install.go:47 +0x94
github.com/spf13/cobra.(*Command).execute(0xc0006b82c0, 0xc0006867b0, 0x3, 0x3, 0xc0006b82c0, 0xc0006867b0)
    /Users/kensipe/projects/go/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:838 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xc0006b8000, 0x0, 0x16ad8c0, 0xc000042118)
    /Users/kensipe/projects/go/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:943 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
    /Users/kensipe/projects/go/pkg/mod/github.com/spf13/cobra@v0.0.7/command.go:883
main.main()
    /Users/kensipe/projects/go/src/github.com/kudobuilder/kudo/cmd/kubectl-kudo/main.go:24 +0x27