search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
139 stars 47 forks source link

Cluster instanciation failure when trying to activate SSL and OIDC on openshift #459

Open looping-aba opened 2 months ago

looping-aba commented 2 months ago

Type of question

Implementation Assistance

Support question

Hello, I succeed to instanciate a 2 node cluster on openshift, but when I tried to activate authentication and SSL :) I'm looking for a workaround... and better a solution ;)

Context

---
apiVersion: nifi.konpyutaika.com/v1
kind: NifiCluster
metadata:
  name: nifi
spec:
  service:
    headlessEnabled: true
    labels:
      cluster-name: nifi
  zkAddress: "<ZOOKEEPER_URL>:2181"
  zkPath: /nifi
  clusterImage: "<HARBOR_URL>/data-fabrique/ucn-nifi:v0.7"
  initContainerImage: 'bash:5.2.2'
  oneNifiNodePerNode: true
  nodeUserIdentityTemplate: "n-%d"
  readOnlyConfig:
    nifiProperties:
      overrideConfigs: |
        nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
        nifi.security.user.oidc.discovery.url=https://<URL_KEYCLOAK>/.well-known/openid-configuration
        nifi.security.user.oidc.client.id=nifi
        nifi.security.user.oidc.client.secret=<CLIENT_SECRET>
        nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
        nifi.security.identity.mapping.value.dn=$1
        nifi.security.identity.mapping.transform.dn=NONE
    bootstrapProperties:
      overrideConfigs: |
        java.arg.2=-Xms2g
        java.arg.3=-Xmx6g
  pod:
    labels:
      cluster-name: nifi
  nodeConfigGroups:
    default_group:
      runAsUser: 1000810000 # set an uid in your namespace range
      fsGroup: 1000810000 # set an gid in your namespace range
      imagePullPolicy: IfNotPresent
      isNode: true
      serviceAccountName: default
      externalVolumeConfigs:
        - name: krb5-config
          mountPath: "/opt/nifi/nifi-current/kerberos"
          configMap:
            name: "krb5-config"
        - name: nifi-keytab
          mountPath: "/opt/nifi/nifi-current/keytabs"
          secret:
            secretName: "nifi-keytab"
      storageConfigs:
        - mountPath: "/opt/nifi/nifi-current/logs"
          name: logs
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
      resourcesRequirements:
        limits:
          cpu: "6"
          memory: 16Gi
        requests:
          cpu: "6"
          memory: 16Gi
  nodes:
    - id: 1
      nodeConfigGroup: "default_group"
    - id: 2
      nodeConfigGroup: "default_group"
  propagateLabels: true
  nifiClusterTaskSpec:
    retryDurationMinutes: 10
  listenersConfig:
    internalListeners:
      - containerPort: 8443
        type: https
        name: https
      - containerPort: 6007
        type: cluster
        name: cluster
      - containerPort: 10000
        type: s2s
        name: s2s
      - containerPort: 9090
        type: prometheus
        name: prometheus
    sslSecrets:
      tlsSecretName: "nifi-secure-test"
      create: true

Behaviour The cluster instanciation fail and no nifi pod is created.

In the certmanager namespace i can see that logs and it seems that cert and issuer are correctly created.

I can see cert in dev01-nifi namespace nifikop-webhook-cert nifi-ca-certificate

I can found issuer in dev01-nifi namespace nifi-issuer nifi-self-signer selfsigned-issuer

I can find secrets in dev01-nifi namespace nifi-1-server-certificate nifi-2-server-certificate nifi-ca-certificate nifi-controller

Logs and Error

I0907 18:00:25.751260       1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="dev01-nifi/nifi-ca-certificate" error="Operation cannot be fulfilled on certificates.cert-manager.io \"nifi-ca-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0907 18:00:25.752202       1 conditions.go:192] Found status change for Certificate "nifi-ca-certificate" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-07 18:00:25.752192024 +0000 UTC m=+696851.285804354
I0907 18:00:25.778058       1 controller.go:162] "cert-manager/certificates-issuing: re-queuing item due to optimistic locking on resource" key="dev01-nifi/nifi-ca-certificate" error="Operation cannot be fulfilled on certificates.cert-manager.io \"nifi-ca-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0907 18:00:25.797666       1 controller.go:162] "cert-manager/certificates-key-manager: re-queuing item due to optimistic locking on resource" key="dev01-nifi/nifi-ca-certificate" error="Operation cannot be fulfilled on certificates.cert-manager.io \"nifi-ca-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0907 18:00:30.257663       1 conditions.go:85] Found status change for Issuer "nifi-issuer" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-07 18:00:30.257649778 +0000 UTC m=+696855.791262085

I have several error regarding the fact client certificate cannont be generated

{"level":"info","time":"2024-09-07T18:00:24.964Z","logger":"controller.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:25.285Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user nifi-controller"}
{"level":"error","time":"2024-09-07T18:00:25.285Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-controller","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi-controller","reconcileID":"8ed91546-8fe3-4013-90bc-49e1cf197af0","error":"**_could not create user certificate: certificates.cert-manager.io \"nifi-controller\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on:_** , <nil>","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:25.289Z","logger":"controller.NifiCluster","caller":"k8sutil/resource.go:51","msg":"resource created","name":"nifi-headless","namespace":"dev01-nifi","kind":""}
{"level":"info","time":"2024-09-07T18:00:25.289Z","logger":"controller.NifiCluster","caller":"controller/nificluster_controller.go:156","msg":"A new resource was not found or may not be ready","reason":"server secret not ready: Secret \"nifi-1-server-certificate\" not found"}
{"level":"info","time":"2024-09-07T18:00:25.290Z","logger":"controller.NifiCluster","caller":"controller/nificluster_controller.go:134","msg":"NifiCluster starting reconciliation","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:25.290Z","logger":"controller.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:25.348Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-1"}
{"level":"error","time":"2024-09-07T18:00:25.348Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-1","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-1","reconcileID":"283ff513-0221-43c0-926f-2f73c70cb546","error":"could not create user certificate: certificates.cert-manager.io \"n-1\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:25.449Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-2"}
{"level":"error","time":"2024-09-07T18:00:25.449Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-2","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-2","reconcileID":"e21d688f-30a7-4bc0-b225-0047e891a871","error":"could not create user certificate: certificates.cert-manager.io \"n-2\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}

And After I can see certificate decoding error

{"level":"error","time":"2024-09-07T18:00:30.649Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nificluster","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiCluster","nifiCluster":{"name":"nifi","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi","reconcileID":"cf2c365e-4f29-4942-8139-06b8fa3936ef","error":"**_failed to decode certificate: failed to decode x509 certificate_** from PEM","errorVerbose":"failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:483\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:139\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiClusterReconciler).Reconcile\n\t/workspace/internal/controller/nificluster_controller.go:146\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:30.755Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user nifi-controller"}
{"level":"error","time":"2024-09-07T18:00:30.755Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-controller","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi-controller","reconcileID":"c594b1f8-fbe1-42d5-9b12-6f7291450248","error":"could not create secret with jks password: secrets \"nifi-controller\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:30.796Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-1"}
{"level":"error","time":"2024-09-07T18:00:30.796Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-1","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-1","reconcileID":"1aa1d201-a684-4385-bfa5-1c9ef3cd5af8","error":"could not create secret with jks password: secrets \"nifi-1-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:30.805Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user n-2"}
{"level":"error","time":"2024-09-07T18:00:30.805Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"n-2","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"n-2","reconcileID":"d9b14cdf-581b-49ce-b635-0ea6699ea018","error":"could not create secret with jks password: secrets \"nifi-2-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:35.770Z","logger":"controller.NifiCluster","caller":"controller/nificluster_controller.go:134","msg":"NifiCluster starting reconciliation","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:35.770Z","logger":"controller.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi"}
{"level":"info","time":"2024-09-07T18:00:35.773Z","logger":"controller.NifiCluster","caller":"controller/controller_common.go:35","msg":"failed to decode certificate: failed to decode x509 certificate from PEM"}
{"level":"error","time":"2024-09-07T18:00:35.773Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nificluster","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiCluster","nifiCluster":{"name":"nifi","namespace":"dev01-nifi"},"namespace":"dev01-nifi","name":"nifi","reconcileID":"b832964d-1306-4685-9711-c39a3872a14f","error":"failed to decode certificate: failed to decode x509 certificate from PEM","errorVerbose":"failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:483\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(*Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:139\ngithub.com/konpyutaika/nifikop/internal/controller.(*NifiClusterReconciler).Reconcile\n\t/workspace/internal/controller/nificluster_controller.go:146\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"info","time":"2024-09-07T18:00:35.894Z","logger":"controller.NifiUser","caller":"controller/controller_common.go:35","msg":"failed to reconcile secret for user nifi-controller"}

NiFiKop version

No response

Golang version

No response

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"c7c6eb21da5c5b9f813ea09a21aa3e7226206993", GitTreeState:"clean", BuildDate:"2023-11-21T17:49:49Z", GoVersion:"go1.19.13 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.10+c79e5e2", GitCommit:"c725f2ce5164bf4165b22d6c28dd0ace4b3b7e9b", GitTreeState:"clean", BuildDate:"2024-02-21T18:19:42Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

NiFi version

1.21 => Image de base de l'image utilisée

forzamehlano commented 1 month ago

We had to add the finalizers permission to the role to get nifikop to work on openshift (well OKD 4.15) when deploying via helm.

Specifically adding additional lines for each of the objects beneath https://github.com/konpyutaika/nifikop/blob/963e3014004cd9fc4fdfe6a931d5e5526f7f5577/helm/nifikop/templates/role.yaml#L82 (nifiusers/finalizers, nificlusters/finalizers and so on)