search

Orange-OpenSource / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://orange-opensource.github.io/nifikop/
Apache License 2.0
128 stars 34 forks source link

NiFi Cluster doesn't spin up #51

Closed VasiliiSurov closed 3 years ago

VasiliiSurov commented 3 years ago

Bug Report

Getting error while deploying simple nifi cluster. {"level":"info","ts":1607960170.9530003,"logger":"cmd","msg":"Operator Version: 0.3.1"} {"level":"info","ts":1607960170.9530435,"logger":"cmd","msg":"Go Version: go1.14.4"} {"level":"info","ts":1607960170.95305,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1607960170.9530544,"logger":"cmd","msg":"Version of operator-sdk: v0.18.1"} {"level":"info","ts":1607960170.9534504,"logger":"leader","msg":"Trying to become the leader."} I1214 15:36:12.004143 1 request.go:621] Throttling request took 1.034656236s, request: GET:https://10.19.240.1:443/apis/scheduling.k8s.io/v1?timeout=32s {"level":"info","ts":1607960172.0770478,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."} {"level":"info","ts":1607960172.077083,"logger":"leader","msg":"Continuing as the leader."} time="2020-12-14T15:36:12Z" level=info msg="Writing ready file." {"level":"info","ts":1607960173.1828148,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"} {"level":"info","ts":1607960173.1836252,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1607960173.1841946,"logger":"cmd","msg":"Starting manager."} {"level":"info","ts":1607960173.1846204,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1607960173.184939,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiregistryclient-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.1851072,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiuser-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.1854053,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiparametercontext-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.1857305,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificlustertask-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.1848311,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.1863284,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifidataflow-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.2862976,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifiparametercontext-controller"} {"level":"info","ts":1607960173.2863894,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifiparametercontext-controller","worker count":1} {"level":"info","ts":1607960173.286246,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifiregistryclient-controller"} {"level":"info","ts":1607960173.2864208,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifiregistryclient-controller","worker count":1} {"level":"info","ts":1607960173.2872162,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifidataflow-controller"} {"level":"info","ts":1607960173.2873166,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifidataflow-controller","worker count":1} {"level":"info","ts":1607960173.2871523,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nificlustertask-controller"} {"level":"info","ts":1607960173.2877488,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nificlustertask-controller","worker count":1} {"level":"info","ts":1607960173.2880201,"logger":"controller_nificlustertask","msg":"Reconciling NifiCluster","Request.Namespace":"nifi","Request.Name":"simplenifi"} {"level":"info","ts":1607960173.2876427,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.2873774,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiuser-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.388855,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.3896422,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifiuser-controller"} {"level":"info","ts":1607960173.4897692,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifiuser-controller","worker count":1} {"level":"info","ts":1607960173.4897482,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1607960173.5903778,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nificluster-controller"} {"level":"info","ts":1607960173.5907698,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nificluster-controller","worker count":1} {"level":"info","ts":1607960173.5911188,"logger":"controller_nificluster","msg":"Reconciling NifiCluster","Request.Namespace":"nifi","Request.Name":"simplenifi"} {"level":"info","ts":1607960173.604301,"logger":"controller_nificluster","msg":"CR status updated","Request.Namespace":"nifi","Request.Name":"simplenifi","status":"ClusterReconciling"} E1214 15:36:13.713654 1 runtime.go:78] Observed a panic: runtime.boundsError{x:1, y:1, signed:true, code:0x0} (runtime error: index out of range [1] with length 1) goroutine 512 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x17436e0, 0xc0006ad1a0) nifikop/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) nifikop/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82 panic(0x17436e0, 0xc0006ad1a0) /usr/local/go/src/runtime/panic.go:969 +0x166 github.com/Orange-OpenSource/nifikop/pkg/util/zookeeper.GetPortAddress(...) nifikop/pkg/util/zookeeper/common.go:32 github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).pod(0xc000945d40, 0xc000000001, 0xc00037e480, 0xc000850c40, 0x1, 0x1, 0x1abca00, 0xc0008dd000, 0x1, 0x1) nifikop/pkg/resources/nifi/pod.go:59 +0x35ad github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).Reconcile(0xc000945d40, 0x1abca00, 0xc0008dd000, 0x15ccee0, 0x1a5fb40) nifikop/pkg/resources/nifi/nifi.go:179 +0xa09 github.com/Orange-OpenSource/nifikop/pkg/controller/nificluster.(ReconcileNifiCluster).Reconcile(0xc0005fd300, 0xc00060a78c, 0x4, 0xc00060a770, 0xa, 0x0, 0xbfedff7b63399626, 0xc000162ea0, 0xc0000c06c8) nifikop/pkg/controller/nificluster/nificluster_controller.go:174 +0x452 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc0002126c0, 0x16ae860, 0xc0009b4a40, 0x17b3c00) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x161 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc0002126c0, 0x203000) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xae sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker(0xc0002126c0) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006f0d80) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006f0d80, 0x1a7b080, 0xc00076a030, 0xc00000fe01, 0xc0001698c0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006f0d80, 0x3b9aca00, 0x0, 0x18e5701, 0xc0001698c0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2 k8s.io/apimachinery/pkg/util/wait.Until(0xc0006f0d80, 0x3b9aca00, 0xc0001698c0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x305 panic: runtime error: index out of range [1] with length 1 [recovered] panic: runtime error: index out of range [1] with length 1

goroutine 512 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) nifikop/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105 panic(0x17436e0, 0xc0006ad1a0) /usr/local/go/src/runtime/panic.go:969 +0x166 github.com/Orange-OpenSource/nifikop/pkg/util/zookeeper.GetPortAddress(...) nifikop/pkg/util/zookeeper/common.go:32 github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).pod(0xc000945d40, 0xc000000001, 0xc00037e480, 0xc000850c40, 0x1, 0x1, 0x1abca00, 0xc0008dd000, 0x1, 0x1) nifikop/pkg/resources/nifi/pod.go:59 +0x35ad github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).Reconcile(0xc000945d40, 0x1abca00, 0xc0008dd000, 0x15ccee0, 0x1a5fb40) nifikop/pkg/resources/nifi/nifi.go:179 +0xa09 github.com/Orange-OpenSource/nifikop/pkg/controller/nificluster.(ReconcileNifiCluster).Reconcile(0xc0005fd300, 0xc00060a78c, 0x4, 0xc00060a770, 0xa, 0x0, 0xbfedff7b63399626, 0xc000162ea0, 0xc0000c06c8) nifikop/pkg/controller/nificluster/nificluster_controller.go:174 +0x452 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc0002126c0, 0x16ae860, 0xc0009b4a40, 0x17b3c00) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x161 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc0002126c0, 0x203000) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xae sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker(0xc0002126c0) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006f0d80) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006f0d80, 0x1a7b080, 0xc00076a030, 0xc00000fe01, 0xc0001698c0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006f0d80, 0x3b9aca00, 0x0, 0x18e5701, 0xc0001698c0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2 k8s.io/apimachinery/pkg/util/wait.Until(0xc0006f0d80, 0x3b9aca00, 0xc0001698c0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x305

What did you do?

! /bin/bash

export GCP_PROJECT=${1} export GCP_ZONE=us-central1-a export CLUSTER_NAME=nifi-cluster

gcloud container clusters create $CLUSTER_NAME \ --cluster-version latest \ --machine-type=e2-medium \ --num-nodes 3 \ --zone $GCP_ZONE \ --project $GCP_PROJECT

gcloud container clusters get-credentials $CLUSTER_NAME \ --zone $GCP_ZONE \ --project $GCP_PROJECT

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)

kubectl create namespace nifi

kubectl create namespace zookeeper

kubectl create namespace cert-manager

cat <<EOF | kubectl apply -f - apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-storage parameters: type: pd-standard provisioner: kubernetes.io/gce-pd reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer EOF

cat <<EOF | kubectl apply -f - apiVersion: v1 kind: ServiceAccount metadata: name: nifikop EOF

helm install nifikop-zk bitnami/zookeeper \ --namespace=nifi \ --set resources.requests.memory=256Mi \ --set resources.requests.cpu=250m \ --set resources.limits.memory=256Mi \ --set resources.limits.cpu=250m \ --set networkPolicy.enabled=true \ --set replicaCount=3 \ --set namespaces={“nifi”}

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/nifikop/master/deploy/crds/v1/nifi.orange.com_nificlusters_crd.yaml kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/nifikop/master/deploy/crds/v1/nifi.orange.com_nifiusers_crd.yaml kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/nifikop/master/deploy/crds/v1/nifi.orange.com_nifiusergroups_crd.yaml kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/nifikop/master/deploy/crds/v1/nifi.orange.com_nifidataflows_crd.yaml kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/nifikop/master/deploy/crds/v1/nifi.orange.com_nifiparametercontexts_crd.yaml kubectl apply -f https://raw.githubusercontent.com/Orange-OpenSource/nifikop/master/deploy/crds/v1/nifi.orange.com_nifiregistryclients_crd.yaml

helm install nifikop \ orange-incubator/nifikop \ --namespace=nifi \ --set namespaces={"nifi"} \ --set resources.requests.memory=256Mi \ --set resources.requests.cpu=250m \ --set resources.limits.memory=256Mi \ --set resources.limits.cpu=250m

cat <<EOF | kubectl create -n nifi -f - apiVersion: nifi.orange.com/v1alpha1 kind: NifiCluster metadata: name: simplenifi spec: service: headlessEnabled: true zkAddress: "nifikop-zk-zookeeper:2181" zkPath: "/simplenifi" clusterImage: "apache/nifi:1.12,1" oneNifiNodePerNode: false nodeConfigGroups: default_group: isNode: true storageConfigs:

What did you expect to see? A clear and concise description of what you expected to happen (or insert a code snippet). NiFi is up and running

What did you see instead? Under which circumstances? A clear and concise description of what you expected to happen (or insert a code snippet).

Environment

kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.9", GitCommit:"4fb7ed12476d57b8437ada90b4f93b17ffaeed99", GitTreeState:"clean", BuildDate:"2020-07-15T16:18:16Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.14-gke.1200", GitCommit:"7c407f5cc8632f9af5a2657f220963aa7f1c46e7", GitTreeState:"clean", BuildDate:"2020-12-01T09:20:59Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

philrogers commented 3 years ago

I experienced the same issue on Kubernetes 1.20 in house Kubernetes

Guys your documentation is good and the idea is great but its pointless posting stuff online and open sourcing it if it fails to function.

Installing the schema fails and after fixing this now the nifikop crashes when you apply simplenificluster.yaml

kubectl apply -f simplenificluster.yaml -n nifi

"level":"info","ts":1608165540.2539074,"logger":"cmd","msg":"Operator Version: 0.3.1"} {"level":"info","ts":1608165540.2540126,"logger":"cmd","msg":"Go Version: go1.14.4"} {"level":"info","ts":1608165540.2540278,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1608165540.2540362,"logger":"cmd","msg":"Version of operator-sdk: v0.18.1"} {"level":"info","ts":1608165540.2545354,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1608165541.2456985,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."} {"level":"info","ts":1608165541.2457423,"logger":"leader","msg":"Continuing as the leader."} time="2020-12-17T00:39:01Z" level=info msg="Writing ready file." {"level":"info","ts":1608165542.1551037,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"} {"level":"info","ts":1608165542.1555028,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1608165542.1561072,"logger":"cmd","msg":"Starting manager."} {"level":"info","ts":1608165542.1570456,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1608165542.157113,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.1572254,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiparametercontext-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.1573653,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificlustertask-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.1574602,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiregistryclient-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.1574664,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifidataflow-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.1575997,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiuser-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.2576873,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nificlustertask-controller"} {"level":"info","ts":1608165542.2578049,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nificlustertask-controller","worker count":1} {"level":"info","ts":1608165542.2578757,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifiregistryclient-controller"} {"level":"info","ts":1608165542.2579699,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifiregistryclient-controller","worker count":1} {"level":"info","ts":1608165542.2581317,"logger":"controller_nificlustertask","msg":"Reconciling NifiCluster","Request.Namespace":"nifi","Request.Name":"simplenifi"} {"level":"info","ts":1608165542.2580483,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.2586732,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifidataflow-controller"} {"level":"info","ts":1608165542.2589414,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifiparametercontext-controller"} {"level":"info","ts":1608165542.2590847,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nifiuser-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.3591578,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.3594947,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nifiuser-controller"} {"level":"info","ts":1608165542.3595445,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifiparametercontext-controller","worker count":1} {"level":"info","ts":1608165542.3596785,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifidataflow-controller","worker count":1} {"level":"info","ts":1608165542.459744,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nifiuser-controller","worker count":1} {"level":"info","ts":1608165542.4598699,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"nificluster-controller","source":"kind source: /, Kind="} {"level":"info","ts":1608165542.5603876,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"nificluster-controller"} {"level":"info","ts":1608165542.5604281,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"nificluster-controller","worker count":1} {"level":"info","ts":1608165542.560541,"logger":"controller_nificluster","msg":"Reconciling NifiCluster","Request.Namespace":"nifi","Request.Name":"simplenifi"} {"level":"info","ts":1608165542.5805182,"logger":"controller_nificluster","msg":"CR status updated","Request.Namespace":"nifi","Request.Name":"simplenifi","status":"ClusterReconciling"} E1217 00:39:02.695793 1 runtime.go:78] Observed a panic: runtime.boundsError{x:1, y:1, signed:true, code:0x0} (runtime error: index out of range [1] with length 1) goroutine 498 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x17436e0, 0xc00066ca80) nifikop/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) nifikop/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82 panic(0x17436e0, 0xc00066ca80) /usr/local/go/src/runtime/panic.go:969 +0x166 github.com/Orange-OpenSource/nifikop/pkg/util/zookeeper.GetPortAddress(...) nifikop/pkg/util/zookeeper/common.go:32 github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).pod(0xc000915b60, 0xc000000001, 0xc000563200, 0xc0000cdc00, 0x1, 0x1, 0x1abca00, 0xc0005dfb40, 0x1, 0x1) nifikop/pkg/resources/nifi/pod.go:59 +0x35ad github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).Reconcile(0xc000915b60, 0x1abca00, 0xc0005dfb40, 0x15ccee0, 0x1a5fb40) nifikop/pkg/resources/nifi/nifi.go:179 +0xa09 github.com/Orange-OpenSource/nifikop/pkg/controller/nificluster.(ReconcileNifiCluster).Reconcile(0xc000773fc0, 0xc00053f4bc, 0x4, 0xc00053f4a0, 0xa, 0x0, 0xbfeec809a167b288, 0xc00086ed80, 0xc00086e758) nifikop/pkg/controller/nificluster/nificluster_controller.go:174 +0x452 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc000562a80, 0x16ae860, 0xc0006c0ec0, 0x17b3c00) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x161 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc000562a80, 0x203000) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xae sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker(0xc000562a80) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009fcac0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0009fcac0, 0x1a7b080, 0xc0009f9a10, 0xc000a04001, 0xc0004ee240) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009fcac0, 0x3b9aca00, 0x0, 0x1, 0xc0004ee240) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2 k8s.io/apimachinery/pkg/util/wait.Until(0xc0009fcac0, 0x3b9aca00, 0xc0004ee240) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x305 panic: runtime error: index out of range [1] with length 1 [recovered] panic: runtime error: index out of range [1] with length 1

goroutine 498 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) nifikop/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105 panic(0x17436e0, 0xc00066ca80) /usr/local/go/src/runtime/panic.go:969 +0x166 github.com/Orange-OpenSource/nifikop/pkg/util/zookeeper.GetPortAddress(...) nifikop/pkg/util/zookeeper/common.go:32 github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).pod(0xc000915b60, 0xc000000001, 0xc000563200, 0xc0000cdc00, 0x1, 0x1, 0x1abca00, 0xc0005dfb40, 0x1, 0x1) nifikop/pkg/resources/nifi/pod.go:59 +0x35ad github.com/Orange-OpenSource/nifikop/pkg/resources/nifi.(Reconciler).Reconcile(0xc000915b60, 0x1abca00, 0xc0005dfb40, 0x15ccee0, 0x1a5fb40) nifikop/pkg/resources/nifi/nifi.go:179 +0xa09 github.com/Orange-OpenSource/nifikop/pkg/controller/nificluster.(ReconcileNifiCluster).Reconcile(0xc000773fc0, 0xc00053f4bc, 0x4, 0xc00053f4a0, 0xa, 0x0, 0xbfeec809a167b288, 0xc00086ed80, 0xc00086e758) nifikop/pkg/controller/nificluster/nificluster_controller.go:174 +0x452 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc000562a80, 0x16ae860, 0xc0006c0ec0, 0x17b3c00) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x161 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc000562a80, 0x203000) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xae sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker(0xc000562a80) nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0009fcac0) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0009fcac0, 0x1a7b080, 0xc0009f9a10, 0xc000a04001, 0xc0004ee240) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0009fcac0, 0x3b9aca00, 0x0, 0x1, 0xc0004ee240) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2 k8s.io/apimachinery/pkg/util/wait.Until(0xc0009fcac0, 0x3b9aca00, 0xc0004ee240) nifikop/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 nifikop/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x305

philrogers commented 3 years ago

I noticed there is several version of the helm chart on https://orange-kubernetes-charts-incubator.storage.googleapis.com/

nifikop-0.1.0.tgz 1592495025947425 1 2020-06-18T15:43:45.947Z "e2a55dbebda9737f4cba744cf1ed2533" 22475 nifikop-0.2.0.tgz 1594986576229367 1 2020-07-17T11:49:36.229Z "009b8fc9a3d225e0414ff719c0cc2ce8" 4572 nifikop-0.2.1.tgz 1597934983363818 1 2020-08-20T14:49:43.363Z "38d7377805e97744c6bafc242915b409" 4664 nifikop-0.3.0.tgz 1600242603490245 1 2020-09-16T07:50:03.490Z "c9b58b84d1e7f43576f37144668c1784" 4697 nifikop-0.3.1.tgz 1603210195154886 1 2020-10-20T16:09:55.154Z "3d8dcdc35b39bdb3d6f3a16f0b4f81d0" 4697 nifikop-0.4.1-alpha.tgz 1605518579651623 1 2020-11-16T09:22:59.651Z "c738434110ea055cdd65d5ffd30c42ab" 27246 nifikop-0.4.2-alpha.tgz 1605526216605204 1 2020-11-16T11:30:16.605Z "b07850dc09d91bc26827b89ef7f19a29" 27256

I tried using the later version nifikop-0.4.2-alpha and it stopped crashing

to do this try helm delete nifikop -n nifi

helm repo add orange-incubator https://orange-kubernetes-charts-incubator.storage.googleapis.com/ kubectl create ns nifi

You have to create the namespace before executing following command

helm install nifikop \ orange-incubator/nifikop \ --namespace=nifi \ --version="0.4.2-alpha" \ --set resources.requests.memory=256Mi \ --set resources.requests.cpu=250m \ --set resources.limits.memory=256Mi \ --set resources.limits.cpu=250m \ --set namespaces={"nifi"} \ --skip-crds

good luck

erdrix commented 3 years ago

Hi, sorry for the problems you are facing when using the operator, the main problem here is actually that you need to explicitly set the version, as Helm only considers versions that look like X.Y.Z to choose the lastest (and here we have the -alpha). This is why the controller crashes (the version deployed by default is an older version, which is not able to manage the new CRDs).

I will publish a documentation update to explicitly add the version in the command line.

Installing the schema fails and after fixing

For the issue about the CRDs, I assume you are talking about the one discussed here : https://github.com/Orange-OpenSource/nifikop/issues/52 and in this PR https://github.com/Orange-OpenSource/nifikop/pull/36.

If true, this problem occurs for newer versions of kubernetes (> 1.17) which adds extra checking, because the CRDs are automatically generated by operator-sdk and the problem is from the definition of a struct from an external library : https://github.com/kubernetes/kubernetes/issues/91395 we have to upgrade the operator-sdk to fix it.