bugslifesolutions commented 2 years ago

Bug Report

What did you do?

Installed nifikop operator using helm chart (single namespace). - verified pod running Also tried with no namespaces, but ran into permission errors on startup due to missing role bindings.
```
NAMESPACE="nifi"
existing=$(kubectl get namespace ${NAMESPACE})
[ "${existing}" = '' ] && kubectl create namespace ${NAMESPACE}
```

You have to create the namespace before executing following command

HELM_EXPERIMENTAL_OCI=1 helm install nifikop \ oci://ghcr.io/konpyutaika/helm-charts/nifikop \ --namespace=${NAMESPACE} \ --version 0.10.0 \ --set image.tag=v0.10.0-release \ --set resources.requests.memory=256Mi \ --set resources.requests.cpu=250m \ --set resources.limits.memory=256Mi \ --set resources.limits.cpu=250m \ --set certManager.clusterScoped=true \ --set namespaces={"${NAMESPACE}"}

helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nifikop nifi 1 2022-03-27 17:53:38.599003529 +0000 UTC deployed nifikop-0.10.0 0.10.0-release


- Applied cert-manager operator using manifests at https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml 

- Created cloudflare-apitoken-secret using this script
```bash
cloudflareSecretName="cloudflare-apitoken-secret";
existingApiToken=$(kubectl -n cert-manager get secret ${cloudflareSecretName} --template='{{.data.apitoken}}' | base64 -d )
if [ "$?" = '0' ]; then
    echo "Secret already exists for $cloudflareSecretName with value $existingApiToken";
fi

read -p "Enter the new https://dash.cloudflare.com/profile/api-tokens API Tokens (or nothing to skip/leave as is $existingApiToken): " secret
if [ $secret != "" ]; then
    kubectl -n cert-manager delete secret $cloudflareSecretName
    kubectl create secret generic -n cert-manager $cloudflareSecretName --from-literal=apitoken=$secret
fi

secret=${secret:-$existingApiToken}
echo "secret is:$secret"
result=$(curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
     -H "Authorization: Bearer $secret" \
     -H "Content-Type:application/json")

echo $result

Applied letsencrypt staging issuer (catch all/no dns filter) - verified certificates were issued

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# disableAccountKeyGeneration: true
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: letsencrypt-staging@bugslifesolutions.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
  # Secret resource that will be used to store the account's private key.
  name: letsencrypt-staging-account-private-key
solvers:
- dns01:
    cloudflare:
      email: admin@bugslifesolutions.com
      apiTokenSecretRef:
        name: cloudflare-apitoken-secret
        key: apitoken

Installed zookeeper - verified pods running

# The bitnami/zookeeper chart templates create 
# fully qualified ZOO_SERVERS hostnames using the
# namespace value.
# eg. 
# - name: ZOO_SERVERS
#   value: zookeeper-0.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::1 zookeeper-1.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::2 zookeeper-2.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::3 
helm template zookeeper bitnami/zookeeper \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set networkPolicy.enabled=true \
--set replicaCount=3 \
--namespace=${NAMESPACE:-tbdNamespace} \
> .target/manifests.yaml

Applied NifiCluster manifest

apiVersion: nifi.konpyutaika.com/v1alpha1
kind: NifiCluster
metadata:
  name: tlsnifi
spec:
  service:
    headlessEnabled: true
  zkAddress: "zookeeper-svc.zookeeper:2181"
  zkPath: "/tlsnifi"
  clusterImage: "apache/nifi:1.12.1"
  oneNifiNodePerNode: true
  managedAdminUsers:
    -  identity : "nifi-admin@bugslifesolutions.com"
       name: "nifiadmin"
  managedReaderUsers:
    -  identity : "nifi-reader@bugslifesolutions.com"
       name: "nifireader"
  propagateLabels: true
  nifiClusterTaskSpec:
    retryDurationMinutes: 10
  nodeConfigGroups:
    default_group:
      isNode: true
      storageConfigs:
        - mountPath: "/opt/nifi/nifi-current/logs"
          name: logs
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/data"
          name: data
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/flowfile_repository"
          name: flowfile-repository
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/nifi-current/conf"
          name: conf
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/content_repository"
          name: content-repository
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/provenance_repository"
          name: provenance-repository
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
      serviceAccountName: "default"
      resourcesRequirements:
        limits:
          cpu: "2"
          memory: 3Gi
        requests:
          cpu: "1"
          memory: 1Gi
  nodes:
    - id: 0
      nodeConfigGroup: "default_group"
    - id: 1
      nodeConfigGroup: "default_group"
  listenersConfig:
    clusterDomain: c2.bugslifesolutions.com
    useExternalDNS: false
    internalListeners:
      - type: "https"
        name: "https"
        containerPort: 8443
      - type: "cluster"
        name: "cluster"
        containerPort: 6007
      - type: "s2s"
        name: "s2s"
        containerPort: 10000
    sslSecrets:
      tlsSecretName: "tlsnifi.c2.bugslifesolutions.com"
      create: true
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: letsencrypt-staging
  externalServices:
    - name: "nifi-cluster"
      spec:
        type: LoadBalancer
        portConfigs:
          - port: 8443
            internalListenerName: "https"   
  readOnlyConfig:
    # NifiProperties configuration that will be applied to the node.
    nifiProperties:
      webProxyHosts:
        - nifi.cluster2.admin.bugslifesolutions.com
        - nifi.c2.admin.bugslifesolutions.com
      # Additionnals nifi.properties configuration that will override the one produced based
      # on template and configurations.
      overrideConfigs: |
        nifi.ui.banner.text=NiFi
        nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration
        nifi.security.user.oidc.client.id=930711295780-i72an91pqj7rib88r23qfv5q7mth8hgv.apps.googleusercontent.com
        nifi.security.user.oidc.client.secret=BvmEyr81P0YXZtIt1FIfGsRs
        nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
        nifi.security.identity.mapping.value.dn=$1
        nifi.security.identity.mapping.transform.dn=NONE

What did you expect to see? A pod created for the NifiCluster.

What did you see instead? Under which circumstances? No pods were created for the NifiCluster and the status.state is ClusterReconciling:

status:
  nodesState:
    "0":
      configurationState: ""
      gracefulActionState:
        actionState: ""
        errorMessage: ""
      initClusterNode: true
      podIsReady: false
  prometheusReportingTask:
    id: ""
    version: 0
  rollingUpgradeStatus:
    errorCount: 0
    lastSuccess: ""
  state: ClusterReconciling

The Nifikop operator logs suggest the secrets aren't created appropriately:

{"level":"info","ts":1648403623.7630396,"logger":"setup","msg":"manager set up with multiple namespaces","namespaces":"nifi"}
...
{"level":"info","ts":1648403625.282073,"logger":"controller.nifidataflow","msg":"Starting workers","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiDataflow","worker count":1}
{"level":"info","ts":1648403877.1215262,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterInitializing"}
{"level":"info","ts":1648403877.1492836,"logger":"controllers.NifiCluster","msg":"Nifi cluster state updated"}
{"level":"info","ts":1648403877.1786754,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterInitialized"}
{"level":"info","ts":1648403877.2189906,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterReconciling"}
{"level":"info","ts":1648403877.2191544,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi"}
{"level":"info","ts":1648403877.3570118,"logger":"controllers.NifiCluster","msg":"resource created","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi","kind":"*v1.Service","name":"tlsnifi-headless"}
{"level":"info","ts":1648403877.3572803,"logger":"controllers.NifiCluster","msg":"A new resource was not found or may not be ready"}
{"level":"info","ts":1648403877.3573396,"logger":"controllers.NifiCluster","msg":"server secret not ready: Secret \"tlsnifi-0-server-certificate\" not found"}
{"level":"info","ts":1648403877.3895488,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterReconciling"}
{"level":"info","ts":1648403877.3896518,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi"}
{"level":"info","ts":1648403877.391266,"logger":"controllers.NifiCluster","msg":"resource updated","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi","kind":"*v1.Service","name":"tlsnifi-headless"}
{"level":"info","ts":1648403877.3913736,"logger":"controllers.NifiCluster","msg":"A new resource was not found or may not be ready"}
{"level":"info","ts":1648403877.3913915,"logger":"controllers.NifiCluster","msg":"server secret not ready: Secret \"tlsnifi-0-server-certificate\" not found"}
{"level":"info","ts":1648403877.6715496,"logger":"controllers.NifiUser","msg":"generated secret not found, may not be ready"}
{"level":"info","ts":1648403877.796963,"logger":"controllers.NifiUser","msg":"failed to reconcile user secret"}
{"level":"error","ts":1648403877.7970812,"logger":"controller.nifiuser","msg":"Reconciler error","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiUser","name":"tlsnifi-0-node.tlsnifi-headless.nifi.svc.c2.bugslifesolutions.com","namespace":"nifi","error":"could not create user certificate: admission webhook \"webhook.cert-manager.io\" denied the request: spec.commonName: Too long: must have at most 64 bytes","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1648403877.7981198,"logger":"controllers.NifiUser","msg":"generated secret not found, may not be ready"}
{"level":"info","ts":1648403877.8216078,"logger":"controllers.NifiUser","msg":"failed to reconcile user secret"}
{"level":"error","ts":1648403877.8217096,"logger":"controller.nifiuser","msg":"Reconciler error","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiUser","name":"tlsnifi-0-node.tlsnifi-headless.nifi.svc.c2.bugslifesolutions.com","namespace":"nifi","error":"could not create secret with jks password: secrets \"tlsnifi-0-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}

The Cert-Manager pod logs indicate an issue with the Domain of the Nifikop tlsnifi-controller owned CertificateRequest:

E0327 18:10:13.819600 1 sync.go:270] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"tlsnifi-controller\": Domain name needs at least one dot" "resource_kind"="Order" "resource_name"="tlsnifi-controller-hjpcs-2838947340" "resource_namespace"="nifi" "resource_version"="v1"

in context:

I0327 18:09:00.095912       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-account-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0327 18:09:05.034646       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-account-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0327 18:10:03.378491       1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Ready" to 2022-03-27 18:10:03.37839815 +0000 UTC m=+63.713073736
I0327 18:10:03.378861       1 trigger_controller.go:181] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="nifi/tlsnifi-controller" "message"="Issuing certificate as Secret does not contain a private key" "reason"="MissingData"
I0327 18:10:03.378913       1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Issuing" to 2022-03-27 18:10:03.378908921 +0000 UTC m=+63.713584482
I0327 18:10:03.762733       1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi-controller" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi-controller\": the object has been modified; please apply your changes to the latest version and try again"
I0327 18:10:03.762897       1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Ready" to 2022-03-27 18:10:03.762888653 +0000 UTC m=+64.097564227
I0327 18:10:12.499043       1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi-controller" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi-controller\": the object has been modified; please apply your changes to the latest version and try again"
I0327 18:10:12.591206       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi-controller-hjpcs" condition "Approved" to 2022-03-27 18:10:12.591190353 +0000 UTC m=+72.925865914
I0327 18:10:12.670590       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi-controller-hjpcs" condition "Ready" to 2022-03-27 18:10:12.670574343 +0000 UTC m=+73.005249913
E0327 18:10:13.819600       1 sync.go:270] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"tlsnifi-controller\": Domain name needs at least one dot" "resource_kind"="Order" "resource_name"="tlsnifi-controller-hjpcs-2838947340" "resource_namespace"="nifi" "resource_version"="v1"
I0327 18:10:13.938765       1 conditions.go:190] Found status change for Certificate "tlsnifi-controller" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2022-03-27 18:10:13.938746383 +0000 UTC m=+74.273421949
I0327 18:10:13.965550       1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="nifi/tlsnifi-controller" "retry_delay"=3599034548569
I0327 18:10:14.023466       1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="nifi/tlsnifi-controller" "retry_delay"=3598976585445

Here is the CertificateRequest:

apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
  annotations:
    cert-manager.io/certificate-name: tlsnifi-controller
    cert-manager.io/certificate-revision: "1"
    cert-manager.io/private-key-secret-name: tlsnifi-controller-s5j8v
  creationTimestamp: "2022-03-27T18:10:12Z"
  generateName: tlsnifi-controller-
  generation: 1
  name: tlsnifi-controller-hjpcs
  namespace: nifi
  ownerReferences:
  - apiVersion: cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: tlsnifi-controller
    uid: 83fecd4e-6f95-45f8-9d9c-612186ce9eb1
  resourceVersion: "23860749"
  uid: c7100e70-6c6e-404a-a9b6-3839d5227795
spec:
  extra:
    authentication.kubernetes.io/pod-name:
    - cert-manager-786b9c87f8-nsb8z
    authentication.kubernetes.io/pod-uid:
    - 035536c1-97a2-49c3-84a8-f64e76c23e98
  groups:
  - system:serviceaccounts
  - system:serviceaccounts:cert-manager
  - system:authenticated
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-staging
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRStEQ0NBdUFDQVFBd0hURWJNQmtHQTFVRUF4TVNkR3h6Ym1sbWFTMWpiMjUwY205c2JHVnlNSUlDSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUF2bWlXLzZvVTNSMmVUK080M25HdmkwT3BoYUdICldHSUNreU5hT0UxNVEwV3dsSmF4dXFpMVZXcG9RTGVBL2liQUpqaEpJWlFMSGErUkpnN3piWWdYUFB2cnkzcG0KRmhjcC94NG95WVBNSFhIdytseFdzYlMvb2l0a2g5OWJMWFA1aWNFRTlGMW9nQWpTNTFCWXpUdDZNWDFYQmdSYwpkZ0RRMUVTdFFaS3p3WTF6UERqSHUwTW90eUh4RXg4bVlmNlUwY1pQa0owYkpDUkdob1dZVUZ5alFMMmROK1QxClY1dDdBdG9zSUhRYW1sTHIxWXZmZy9OYVljRHNOWjNIaWVUTTZNM1NsWWZ3TlE5eTFzMTRWVks2Nzl4bEJJUFcKd0xHYWo5SmVmZWZsa0V0MXQ3a2xqK2kvRUZGb05xcERQOXVFajhFTVJsNjA5S2xRYzkzUC9vaTdqUi9TdXFZdwp1Q1NWeVJBVElSMjd5eSsvU0VSUnRuYnc0Y3NHbkJ0SFA4czVlMTdxVGdkak02QjZLblFTWSs4NEhtUFh1cGRGCjBuZDhWUXFsZTl6a2hYVzh1VTN2K1k3RXZUdytUTVVLTDFjZTBsdDNxaG45ZkpsYUsweE8yT2w3UFk2N2xjcGMKdG9KdlBDSXhBV2doMXEwQkE4ZzJyV0gzRGV3VzhuZDYvMmNyN241ZXA3ZUl5aUlLSHFBb3VkUDBENDJ3UlRSUwpsaFYyUlV1VnZ4aUplUmdtbWp2VCtXWi9PWnRUa2ZNUkJCVjNtYUs4VUNOYzBUejhReWhzdWpnejIzcU5vakt4CjBZNHgvWFlhYVE1MitZY1ZUUE9IWHdCdmwwWUVzb0h4M08ydDBuaWI4Tk5BelRVbzIwWHgweXhDOC9yZ3gzSEcKVDdsaHRpcExpUUFBc3JVQ0F3RUFBYUNCbFRDQmtnWUpLb1pJaHZjTkFRa09NWUdFTUlHQk1GTUdBMVVkRVFSTQpNRXFDRW5Sc2MyNXBabWt0WTI5dWRISnZiR3hsY29ZMGMzQnBabVpsT2k4dmRHeHpibWxtYVM5dWN5OXVhV1pwCkwyNXBabWwxYzJWeUwzUnNjMjVwWm1rdFkyOXVkSEp2Ykd4bGNqQUxCZ05WSFE4RUJBTUNBQUF3SFFZRFZSMGwKQkJZd0ZBWUlLd1lCQlFVSEF3SUdDQ3NHQVFVRkJ3TUJNQTBHQ1NxR1NJYjNEUUVCRFFVQUE0SUNBUUNxUGxBTApNaGRSZzBRaVFtNzQwKy9GQXJicjRqUjVoSWZXelhXN1pON0xqbXBmMkJSZlpnbUNHOFdRQVMzWWtQeXFicVdOCitubTcwYWRZdVAweHNFTHEwRi9aL096R05kU0J0OEdVSHJ2di9NalYzVUl4bjVKUDgyYTBVcXpibXU5VVhOMHgKMXFnOElQcDVuMTBoSUJmWDViK1Jabm9pYnowcmtjcVNqMjUyTnZZd3lkNmt6cTV2ejRnQ2hjVW9CZEhOeXBkMgphL0dxQWkrYm1RUSswanJyejJ6dUppNC9JUlF5eTNCT1FSRW5tVDJCMlh5YVhtQnJtZ1hmZytTbGdzVWN0NEFhClE2MXBFQUVKT3F3aWtVbmZXbndqcnh5Y0xoYkR0aGo0R0drZUgzdmplbWtkbENBMWdxSTJGVmU5WWJpWkt6KzIKNlg4bGxRNlAzaDhyY1pNbVRUcUl1eldIMFEzZVBUWTdtZGt1YW1hR1h2TisvM3hkTlNiaFZuckwycHNyTFFXUwpxM096aXB1WFM5UFAySDk2cVplSy8wWlpDWFhpNXc4czlPZW1RdUtFcXF3RkJIYXdSckY1SE5UTWk1ekVuTjZpCm1xZ2trNmdZZmk0L2RFRFdSeEJ6enFxMHdPOThRUkxsUUkxOVQ5R1JzMk1WaXpBM0pNUUNlcHRnVlR3R3lDUnkKN3pqNkQwUEZ5Z0gvMmt5RGlsZ0NGcUVSYkd1K2R3RmhmejhFNUIwVElmVC9DSXlhYWtmMFA4N3VQVnp4N0hEVwpCaDZEckxmbWRlRDNjR1htL2tPT09NT21OaWkyZ0FFVzh3RTBkQWdRam9BbkFSUE1VMDNIcE5QeFJhQXVmZXFTCmJ4cm5CdmZtSHhrVnlwd01abjJDYTJQVEZvbmxOREozZk1hYmtBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
  uid: 62f7108a-f2ba-4b63-b558-e52ec01a78aa
  usages:
  - client auth
  - server auth
  username: system:serviceaccount:cert-manager:cert-manager
status:
  conditions:
  - lastTransitionTime: "2022-03-27T18:10:12Z"
    message: Certificate request has been approved by cert-manager.io
    reason: cert-manager.io
    status: "True"
    type: Approved
  - lastTransitionTime: "2022-03-27T18:10:12Z"
    message: 'Failed to wait for order resource "tlsnifi-controller-hjpcs-2838947340"
      to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:rejectedIdentifier:
      Error creating new order :: Cannot issue for "tlsnifi-controller": Domain name
      needs at least one dot'
    reason: Failed
    status: "False"
    type: Ready
  failureTime: "2022-03-27T18:10:13Z"

Environment

nifikop version:

apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: privileged
creationTimestamp: "2022-03-27T17:53:40Z"
generateName: nifikop-5d7d6fb5cd-
labels:
app: nifikop
control-plane: nifikop
name: nifikop
operator: nifi
pod-template-hash: 5d7d6fb5cd
release: nifikop
name: nifikop-5d7d6fb5cd-trjll
namespace: nifi
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: nifikop-5d7d6fb5cd
uid: 4e13caad-5cae-47f4-82fc-93b1b391beec
resourceVersion: "23854732"
uid: 8b09a927-397d-432b-a6d5-7a0264e6324e
spec:
containers:
- args:
- metrics-bind-address=8081
- --leader-elect
- --cert-manager-enabled=true
command:
- /manager
env:
- name: WATCH_NAMESPACE
  value: nifi
- name: POD_NAME
  valueFrom:
    fieldRef:
      apiVersion: v1
      fieldPath: metadata.name
- name: OPERATOR_NAME
  value: nifikop
- name: LOG_LEVEL
  value: Info
image: ghcr.io/konpyutaika/docker-images/nifikop:v0.10.0-release
imagePullPolicy: Always
livenessProbe:
  failureThreshold: 3
  httpGet:
    path: /healthz
    port: 8081
    scheme: HTTP
  initialDelaySeconds: 15
  periodSeconds: 20
  successThreshold: 1
  timeoutSeconds: 1
name: nifikop
ports:
- containerPort: 8081
  name: metrics
  protocol: TCP
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /readyz
    port: 8081
    scheme: HTTP
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
resources:
  limits:
    cpu: 250m
    memory: 256Mi
  requests:
    cpu: 250m
    memory: 256Mi
securityContext:
  allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: kube-api-access-rh9rl
  readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: worker5
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsUser: 1000
serviceAccount: nifikop
serviceAccountName: nifikop
terminationGracePeriodSeconds: 10
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-rh9rl
projected:
  defaultMode: 420
  sources:
  - serviceAccountToken:
      expirationSeconds: 3607
      path: token
  - configMap:
      items:
      - key: ca.crt
        path: ca.crt
      name: kube-root-ca.crt
  - downwardAPI:
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
        path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:40Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:50Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:50Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:40Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://e9836ba7f894653319d3d8fed0549270b6323083f04330b54b909cb775a7b782
image: ghcr.io/konpyutaika/docker-images/nifikop:v0.10.0-release
imageID: ghcr.io/konpyutaika/docker-images/nifikop@sha256:99a59041be7cf06e69f4112b9e1e2c6ee6a0320a7942680bc7daf1d13da0a6be
lastState: {}
name: nifikop
ready: true
restartCount: 0
started: true
state:
  running:
    startedAt: "2022-03-27T17:53:43Z"
hostIP: 10.5.2.61
phase: Running
podIP: 10.244.12.80
podIPs:
- ip: 10.244.12.80
qosClass: Guaranteed
startTime: "2022-03-27T17:53:40Z"

go version:

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:
NiFi version: apache/nifi:1.12.1

Possible Solution Suffix the auto-created controller Common Name/Domain with the NifiCluster's spec.listenersConfig.clusterDomain eg. tlsnifi-controller.c2.bugslifesolutions.com Pattern: <metadata.name>-controller.<listenersConfig.clusterDomain>

Additional context

bugslifesolutions commented 2 years ago

https://github.com/konpyutaika/nifikop/blob/b9f6cec46568e1ea6a3243f9b0c0565ffea93050/api/v1alpha1/nificluster_types.go#L711 Seems to have a work-around via setting Spec.NifiControllerTemplate to a value.

I'll give it a try...

bugslifesolutions commented 2 years ago

The override worked as expected..

I0327 23:01:31.775293       1 trigger_controller.go:181] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="nifi/tlsnifi.c2.bugslifesolutions.com" "message"="Issuing certificate as Secret does not contain a private key" "reason"="MissingData"
I0327 23:01:31.775286       1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi.c2.bugslifesolutions.com" condition "Ready" to 2022-03-27 23:01:31.775224688 +0000 UTC m=+17552.109900271
I0327 23:01:31.775427       1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi.c2.bugslifesolutions.com" condition "Issuing" to 2022-03-27 23:01:31.775418876 +0000 UTC m=+17552.110094455
I0327 23:01:32.196356       1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi.c2.bugslifesolutions.com" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi.c2.bugslifesolutions.com\": the object has been modified; please apply your changes to the latest version and try again"
I0327 23:01:32.196503       1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi.c2.bugslifesolutions.com" condition "Ready" to 2022-03-27 23:01:32.19649708 +0000 UTC m=+17552.531172641
I0327 23:01:37.111923       1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi.c2.bugslifesolutions.com" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi.c2.bugslifesolutions.com\": the object has been modified; please apply your changes to the latest version and try again"
I0327 23:01:37.201653       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi.c2.bugslifesolutions.com-kq4qb" condition "Approved" to 2022-03-27 23:01:37.201638867 +0000 UTC m=+17557.536314430
I0327 23:01:37.273872       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi.c2.bugslifesolutions.com-kq4qb" condition "Ready" to 2022-03-27 23:01:37.273860197 +0000 UTC m=+17557.608535757

ntlzthm8 commented 2 months ago

@bugslifesolutions how did you fix the problem?

mh013370 commented 2 months ago

@bugslifesolutions how did you fix the problem?

You can influence the identities that nifikop assigns to the controller (the identity for the operator itself) and nifi nodes via various NifiCluster properties, all of which are documented here: https://konpyutaika.github.io/nifikop/docs/v1.9.0/5_references/1_nifi_cluster/

Sometimes the default template can result in invalid names (usually too long in my experience), so overriding these templates is how you fix that.

Specifically, see:

nifiControllerTemplate
nodeUserIdentityTemplate

konpyutaika / nifikop

secure cluster with cert-manager Letsencrypt issuer: controller certificate has invalid Common Name value #77

Bug Report

You have to create the namespace before executing following command