"error":"could not create secret with jks password: secrets \"simplenifisecure-1-server-certificate\" already exists"

[narayanbhawar10]

What steps will reproduce the bug?

sslSecrete create true
sslSecrets: tlsSecretName: "test-nifikop" create: true

What is the expected behavior?

Nifi cluster is not getting schedule.

What do you see instead?

Nifi cluster is not getting schedule.

Possible solution

No response

NiFiKop version

latest

Golang version

latest

Kubernetes version

latest

NiFi version

No response

Additional context

No response

[Demcheck]

Hello everyone. I have same problem.

[Demcheck]

I have used your new helm chart to deploy nifi-cluster but unfortunately when I tried set it up with ssl enabled and it has not worked. In logs I have next messages:

{"level":"info","time":"2023-03-31T04:39:43.069Z","logger":"controllers.NifiUser","caller":"controllers/controller_common.go:34","msg":"failed to reconcile secret for user nifi-cluster-3-node.nifi-cluster-headless.myns.svc.cluster.local"} {"level":"error","time":"2023-03-31T04:39:43.069Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-cluster-3-node.nifi-cluster-headless.myns.svc.cluster.local","namespace":"myns"},"namespace":"myns","name":"nifi-cluster-3-node.nifi-cluster-headless.myns.svc.cluster.local","reconcileID":"10515e39-4ff0-43d2-9e84-2423a6c7b0cd","error":"could not create secret with jks password: secrets \"nifi-cluster-3-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"} {"level":"info","time":"2023-03-31T04:39:46.932Z","logger":"controllers.NifiUser","caller":"controllers/controller_common.go:34","msg":"failed to reconcile secret for user nifi-cluster-controller"} {"level":"error","time":"2023-03-31T04:39:46.932Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-cluster-controller","namespace":"myns"},"namespace":"myns","name":"nifi-cluster-controller","reconcileID":"a95d7f35-aeb2-480d-a359-f44ca9c8a30b","error":"error checking controller reference on user secret: Object myns/nifi-cluster-controller is already owned by another Certificate controller nifi-cluster-controller","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}

[narayanbhawar10]

@mh013370 Can you help Please.

[mh013370]

@Demcheck : The error you're encountering is in the logs you shared.

"could not create secret with jks password: secrets "nifi-cluster-3-server-certificate" already exists"

Delete nifi-cluster-3-server-certificate secret and nifikop will auto-retry the secret creation.

@narayanbhawar10 I need more information about your setup. Do you have logs from nifikop? It will usually log why it's hung up on something.

[Demcheck]

@mh013370 I did it but it has not helped. Operator has created new certififactes and I still have errors in logs:

{"level":"info","time":"2023-03-31T08:57:51.048Z","logger":"controllers.NifiUser","caller":"controllers/controller_common.go:34","msg":"failed to reconcile secret for user nifi-cluster-1-node.nifi-cluster-headless.myns.svc.cluster.local"} {"level":"error","time":"2023-03-31T08:57:51.048Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-cluster-1-node.nifi-cluster-headless.myns.svc.cluster.local","namespace":"myns"},"namespace":"myns","name":"nifi-cluster-1-node.nifi-cluster-headless.myns.svc.cluster.local","reconcileID":"8abc0ef5-f46a-432e-b458-3d6f286218f5","error":"could not create secret with jks password: secrets \"nifi-cluster-1-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"} {"level":"info","time":"2023-03-31T08:57:51.054Z","logger":"controllers.NifiUser","caller":"controllers/controller_common.go:34","msg":"failed to reconcile secret for user nifi-cluster-2-node.nifi-cluster-headless.myns.svc.cluster.local"} {"level":"error","time":"2023-03-31T08:57:51.054Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-cluster-2-node.nifi-cluster-headless.myns.svc.cluster.local","namespace":"myns"},"namespace":"myns","name":"nifi-cluster-2-node.nifi-cluster-headless.myns.svc.cluster.local","reconcileID":"3a7dce11-8c36-45df-83e0-1e76778c065a","error":"could not create secret with jks password: secrets \"nifi-cluster-2-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"} {"level":"info","time":"2023-03-31T08:57:51.061Z","logger":"controllers.NifiUser","caller":"controllers/controller_common.go:34","msg":"failed to reconcile secret for user nifi-cluster-3-node.nifi-cluster-headless.myns.svc.cluster.local"} {"level":"error","time":"2023-03-31T08:57:51.061Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-cluster-3-node.nifi-cluster-headless.myns.svc.cluster.local","namespace":"myns"},"namespace":"myns","name":"nifi-cluster-3-node.nifi-cluster-headless.myns.svc.cluster.local","reconcileID":"b68b6878-948a-45eb-bc03-2e1bc902feae","error":"could not create secret with jks password: secrets \"nifi-cluster-3-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"} {"level":"info","time":"2023-03-31T08:57:55.818Z","logger":"controllers.NifiUser","caller":"controllers/controller_common.go:34","msg":"failed to reconcile secret for user nifi-cluster-controller"} {"level":"error","time":"2023-03-31T08:57:55.818Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nifiuser","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiUser","nifiUser":{"name":"nifi-cluster-controller","namespace":"myns"},"namespace":"myns","name":"nifi-cluster-controller","reconcileID":"260c2848-f862-481c-a665-9370212bd30f","error":"error checking controller reference on user secret: Object myns/nifi-cluster-controller is already owned by another Certificate controller nifi-cluster-controller","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"} {"level":"info","time":"2023-03-31T08:58:05.912Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:121","msg":"NifiCluster starting reconciliation","clusterName":"nifi-cluster"} {"level":"info","time":"2023-03-31T08:58:05.912Z","logger":"controllers.NifiCluster","caller":"certmanagerpki/certmanager_pki.go:83","msg":"Reconciling cert-manager PKI","clusterName":"nifi-cluster"} {"level":"info","time":"2023-03-31T08:58:05.913Z","logger":"controllers.NifiCluster","caller":"controllers/controller_common.go:34","msg":"failed to decode certificate: Failed to decode x509 certificate from PEM"} {"level":"error","time":"2023-03-31T08:58:05.913Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nificluster","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiCluster","nifiCluster":{"name":"nifi-cluster","namespace":"myns"},"namespace":"myns","name":"nifi-cluster","reconcileID":"f0300c88-18f3-4405-9b97-8ecd7c6c9fde","error":"failed to decode certificate: Failed to decode x509 certificate from PEM","errorVerbose":"Failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:455\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:139\ngithub.com/konpyutaika/nifikop/controllers.(NifiClusterReconciler).Reconcile\n\t/workspace/controllers/nificluster_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}

[mh013370]

Did you look at cert-manager logs to see if there are issues there?

And did you follow the guide here? https://konpyutaika.github.io/nifikop/docs/v1.0.0/3_manage_nifi/1_manage_clusters/1_deploy_cluster/4_ssl_configuration

[narayanbhawar10]

We are following above guide and We are not getting any error logs in cert-manager. We have deleted the secret for which it is giving error and reinstall the nificluster but it is giving same error.

[Demcheck]

@mh013370 Hello. Do you have any updates? I tried defferent configurations but nothing worked.

[narayanbhawar10]

@mh013370 It would be helpful if you share us steps to enable ssl in nifi cluster ,after following doc we are facing issues. Thanks in Advance.

[r65535]

I'm able to get SSL working by following the quick start guide, followed by applying this sample NiFiCluster YAML (with basic tweaks, for OIDC etc)

Are you able to share the NiFiCluster YAML you're trying to apply? Minus anything sensitive

[narayanbhawar10]

@r65535 Thanks for responding , can you share please your sample yaml to enable ssl and odic configuration which are working for you it would be helpful.

[r65535]

It's identical to the one I linked above, but with different admin users and different OIDC values

[narayanbhawar10]

{"level":"error","time":"2023-05-02T10:39:47.623Z","caller":"controller/controller.go:326","msg":"Reconciler error","controller":"nificluster","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiCluster","nifiCluster":{"name":"simplenifisecure","namespace":"nifi"},"namespace":"nifi","name":"simplenifisecure","reconcileID":"ddb76281-9569-4b21-95d9-83f0c81848b8","error":"failed to decode certificate: Failed to decode x509 certificate from PEM","errorVerbose":"Failed to decode x509 certificate from PEM\nfailed to decode certificate\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(Reconciler).getServerAndClientDetails\n\t/workspace/pkg/resources/nifi/nifi.go:455\ngithub.com/konpyutaika/nifikop/pkg/resources/nifi.(Reconciler).Reconcile\n\t/workspace/pkg/resources/nifi/nifi.go:139\ngithub.com/konpyutaika/nifikop/controllers.(NifiClusterReconciler).Reconcile\n\t/workspace/controllers/nificluster_controller.go:133\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234"}

[narayanbhawar10]

@r65535 Can we debug something by above error that is Failed to decode x509 certificate from PEM","errorVerbose":"Failed to decode x509 certificate from PEM\nfailed to decode . I could not proceed and tried all possibilities.

[Demcheck]

@narayanbhawar10 I have found some bugs in code and fixed it. After testing I will create pull request.

[r65535]

I think I've seen this before when the cert-manager certificate DN is too long so can't generate a valid cert.

Are you able to try and add nodeUserIdentityTemplate to your NiFiCluster spec? Something like: nodeUserIdentityTemplate: "n-%d"

You might need to completely delete everything related to the old deployment before trying this change

[narayanbhawar10]

Hello @r65535 as @Demcheck commented that there is some bug in code and fixed.After testing they will confirm so I am waiting for their response let's see if it works. In parallel I will work on config you have provided in above.

[narayanbhawar10]

@r65535 thank you so much for your help I really appreciate it worked after adding property nodeUserIdentityTemplate: "n-%d" .

[narayanbhawar10]

One more thing @r65535 NiFi cluster is started with ssl/https but when I am using nifi api i.e. https://simplenifisecure-headless.nifi.svc.uhn7kls16.local:8443/nifi-api/controller/cluster has completed I am getting bad certificate issue ,could you please help how I can resolve this issue . I have following configurations and I have not added OIDC OR LDAP configs.

    nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
    nifi.security.identity.mapping.value.dn=$1
    nifi.security.identity.mapping.transform.dn=NONE

Is this issue occurring because I missed OIDC configs?

[r65535]

Which authentication method are you using? Whatever has been picked, needs to be used when making calls to the NiFi API. (e.g. if you're using OIDC, a token must be provided in the HTTP call).

This isn't a NiFiKop-specific thing. This is the NiFi app ensuring anonymous users can't interact with the API 😄

[narayanbhawar10]

Thanks,but is there any option to enable single user authentication in nifi cluster yaml.As of now I don't want to use OIDC authentication.

[narayanbhawar10]

@r65535 can you help in above query.

[r65535]

I don't think this can be configured through NiFiKop currently. I'm doing some testing locally, but will probably have to raise a pull request with some changes to enable it 😄