search

Orange-OpenSource / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://orange-opensource.github.io/nifikop/
Apache License 2.0
129 stars 34 forks source link

Deploying Secure Cluster on AKS #21

Open borkod opened 4 years ago

borkod commented 4 years ago

Bug Report

Hello. This is a very interesting project 👍

I am trying to follow https://orange-opensource.github.io/nifikop/blog/secured_nifi_cluster_on_gcp/ , but deploy it on Azure Kubernetes Service.

I've deployed:

I've updated the nifi cluster resource yaml file with appropriate values from above.

When I try to deploy it, I don't see any pod resources even created.

Any suggestions? What's the best way to debug why no pods are even being created? kubectl describe on the nificluster resource doesn't provide any useful information.

I was able to deploy a working cluster on AKS using simple nifi cluster sample (not secured).

Thanks for any suggestions and help!

erdrix commented 4 years ago

Hi, Thanks to you for trying it :D

Could you share your NifiCluster resource ?

One issue could be that the certificates are not ready (in the NiFiKop pod you should have some logs which say if it's the case).

If you check for Certificate and CertificateRequest resources you may have information into the description !

One classic error is that the common name is longer than 64 bits (we are thinking of a way to work around this case !).

borkod commented 4 years ago

Hi,

Here is the NiFiCluster resource:

apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
  name: nifi
  namespace: nifi
spec:
  service:
    headlessEnabled: true
  zkAddresse: "zknifi-zookeeper-headless.default.svc.cluster.local:2181"
  zkPath: "/sec"
  clusterImage: "apache/nifi:1.11.4"
  clusterSecure: true
  siteToSiteSecure: true
  oneNifiNodePerNode: false
  initialAdminUser: Borko@myfakeemail.com
  propagateLabels: true
  nifiClusterTaskSpec:
    retryDurationMinutes: 10
  readOnlyConfig:
    # NifiProperties configuration that will be applied to the node.
    nifiProperties:
      webProxyHosts:
        - domain:8443
      # Additionnals nifi.properties configuration that will override the one produced based
      # on template and configurations.
      overrideConfigs: |
        nifi.security.user.oidc.discovery.url=https://sso.azure.cloud.blahblah/.well-known/openid-configuration
        nifi.security.user.oidc.client.id=nifi-aks
        nifi.security.user.oidc.client.secret=s3cr3t
        nifi.security.identity.mapping.pattern.dn=CN=(.*)(?:, (?:O|OU)=.*)?
        nifi.security.identity.mapping.value.dn=$1
        nifi.security.identity.mapping.transform.dn=NONE
  nodeConfigGroups:
    default_group:
      isNode: true
      storageConfigs:
        - mountPath: "/opt/nifi/nifi-current/logs"
          name: logs
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "nifi-ssd-wait"
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/data"
          name: data
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "nifi-ssd-wait"
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/flowfile_repository"
          name: flowfile-repository
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "nifi-ssd-wait"
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/nifi-current/conf"
          name: conf
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "nifi-ssd-wait"
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/content_repository"
          name: content-repository
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "nifi-ssd-wait"
            resources:
              requests:
                storage: 10Gi
        - mountPath: "/opt/nifi/provenance_repository"
          name: provenance-repository
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "nifi-ssd-wait"
            resources:
              requests:
                storage: 10Gi
      serviceAccountName: "default"
      resourcesRequirements:
        limits:
          cpu: "2"
          memory: 3Gi
        requests:
          cpu: "1"
          memory: 1Gi
  nodes:
    - id: 0
      nodeConfigGroup: "default_group"
    - id: 1
      nodeConfigGroup: "default_group"
    - id: 2
      nodeConfigGroup: "default_group"
  listenersConfig:
    useExternalDNS: false
    # clusterDomain: <domain>
    internalListeners:
      - type: "https"
        name: "https"
        containerPort: 8443
      - type: "cluster"
        name: "cluster"
        containerPort: 6007
      - type: "s2s"
        name: "s2s"
        containerPort: 10000
    sslSecrets:
      tlsSecretName: "test-nifikop"
      create: true
      issuerRef:
        name: letsencrypt-nifi
        kind: Issuer

I am using Let's Encrypt Issuer:

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: letsencrypt-nifi
  namespace: nifi
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: myemail
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-nifi
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
      - http01:
          ingress:
            ingressTemplate:
              metadata:
                annotations:
                  "external-dns.alpha.kubernetes.io/ttl": "5"

Here's the storage class for completeness:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: nifi-ssd-wait
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  storageaccounttype: StandardSSD_LRS
  kind: Managed

I installed cert-manager using: 

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.2/cert-manager.yaml

I'll try again this afternoon and inspect the logs as you suggest.

Thanks for the suggestions and quick reply!

erdrix commented 4 years ago

Hum I think that your trouble comes from let's encrypt because you don't use an external dns with a supported dns host ...

With this configuration your hostname will be with cluster.local suffix, which will be rejected by let's encrypt issuer. If you want to use it you need to set an external dns as I did with cloud DNS in the blog post. Otherwise you can remove the let's encrypt issuer and just use a self signed issuer (default configuration if nothing is specified).

borkod commented 4 years ago

I am using using Azure DNS zone with a valid DNS entry.

I initially tried the external host walk-through (https://orange-opensource.github.io/nifikop/blog/2020-06-30-secured_nifi_cluster_on_gcp_with_external_dns/), but that didn't work neither, so I decided to try it without the external dns. In that case, no nifi pods were created. I did see acme solver pods created though.

I'll try the External DNS option again and see if there are any logs in the NiFiKop pod or if the certificates get created properly.

Thanks!

borkod commented 4 years ago

Hello,

Quick update.

I figured out where I was going wrong. I didn't understand exactly how the whole thing is supposed to work (I'm relatively new to operators and NiFi). I removed the let's encrypt issuer and deployed a self-signed issuer. The pods get deployed:

image

However, trying to access the service via the external loadbalancer IP (40.82.170.144:8443/nifi) results in a timeout.

Here are the logs from one of the pods:

❯ kubectl logs nifi-0-nodeq6r8n -n nifi
Waiting for host to be reachable
failed to reach nifi-0-node.nifi-headless.nifi.svc.cluster.local:8443
Found: , expecting: 100.64.1.122
Found : 100.64.1.122
Ip match for 100.64.1.122
Hostname is successfully binded withy IP adress
* Expire in 0 ms for 6 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 0 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 1 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Could not resolve host: nifi-headless.nifi.svc.cluster.local
* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Closing connection 0
curl: (6) Could not resolve host: nifi-headless.nifi.svc.cluster.local

Java home: /usr/local/openjdk-8
NiFi home: /opt/nifi/nifi-current

Bootstrap Config File: /opt/nifi/nifi-current/conf/bootstrap.conf

2020-07-23 19:53:22,599 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi...
2020-07-23 19:53:22,599 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /opt/nifi/nifi-current
2020-07-23 19:53:22,599 INFO [main] org.apache.nifi.bootstrap.Command Command: /usr/local/openjdk-8/bin/java -classpath /opt/nifi/nifi-current/./conf:/opt/nifi/nifi-current/./lib/jetty-schemas-3.1.jar:/opt/nifi/nifi-current/./lib/log4j-over-slf4j-1.7.30.jar:/opt/nifi/nifi-current/./lib/nifi-properties-1.11.4.jar:/opt/nifi/nifi-current/./lib/nifi-runtime-1.11.4.jar:/opt/nifi/nifi-current/./lib/logback-core-1.2.3.jar:/opt/nifi/nifi-current/./lib/logback-classic-1.2.3.jar:/opt/nifi/nifi-current/./lib/nifi-framework-api-1.11.4.jar:/opt/nifi/nifi-current/./lib/nifi-api-1.11.4.jar:/opt/nifi/nifi-current/./lib/slf4j-api-1.7.30.jar:/opt/nifi/nifi-current/./lib/javax.servlet-api-3.1.0.jar:/opt/nifi/nifi-current/./lib/nifi-nar-utils-1.11.4.jar:/opt/nifi/nifi-current/./lib/jcl-over-slf4j-1.7.30.jar:/opt/nifi/nifi-current/./lib/jul-to-slf4j-1.7.30.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Djava.security.egd=file:/dev/urandom -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/opt/nifi/nifi-current/./conf/nifi.properties -Dnifi.bootstrap.listen.port=40045 -Dapp=NiFi -Dorg.apache.nifi.bootstrap.config.log.dir=/opt/nifi/nifi-current/logs org.apache.nifi.NiFi
2020-07-23 19:53:22,658 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 38

I don't really understand the nifi-0-node.nifi-headless.nifi.svc.cluster.local in the logs. The pod is not nifi-0-node. Seems like maybe the FQDN is not being resolved correctly somewhere? Or am I just doing something wrong again 😝

I'll look into it and try to debug a bit more, but any insight would be greatly appreciated. Again, thanks!!!

clettieri commented 4 years ago

I too have been getting the same log output on my NiFi PODs. When I let nifikop create the certificates, using curl from the NiFi POD shows a bad certificate.

curl -k https://nifi-cluster-headless.nifi.svc.cluster.local:8443 -vvvvv

* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, bad certificate (554):
* error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
* Closing connection 0
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

EDIT: After looking into this further it seems I need to curl with the client certificate to the server. Perhaps this has something to do with why the PODs can't connect to the service as well. I'm still sorting out a few things on my end regarding OpenID connect credentials. Will update here when I can.

UPDATE: I still get this same error in borkod's log - unable to resolve host - when trying to run the unsecured cluster using the sample config.

clettieri commented 4 years ago

I managed to get the unsecured Cluster working and could connect via the loadbalancer. I needed to remove the sslSecrets from my config all together so the readiness check on the POD would complete.

However my logs still showed

* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Closing connection 0
curl: (6) Could not resolve host: nifi-headless.nifi.svc.cluster.local
erdrix commented 4 years ago

I don't really understand the nifi-0-node.nifi-headless.nifi.svc.cluster.local in the logs. The pod is not nifi-0-node. Seems like maybe the FQDN is not being resolved correctly somewhere

Yes your pod is not nifi-0-node, however if you go inside your container and run the command hostname, it will return nifi-0-node, because inside the operator we override the hostname : https://github.com/Orange-OpenSource/nifikop/blob/master/pkg/resources/nifi/pod.go#L323

For the error : 

* Expire in 2 ms for 1 (transfer 0x555c16d7bf50)
* Closing connection 0
curl: (6) Could not resolve host: nifi-headless.nifi.svc.cluster.local

I think I will improve the log to avoid confusion ^^ and maybe add some documentation :D

To sum up, when the NiFi container starts, the first thing we try is to reach the NiFi cluster to get the cluster node list.

In case the node is an initial node (you can find them by making a kubectl describe nificluster <nificluster name>, in the NifiCluster.Status field) this is not necessary to be able to join the cluster, because we are sure that it is already a node in the cluster (which is your case, even if you have this error the NiFi node is running).

When we add a new node before starting NiFi, we need to check that the node is not part of the cluster, that's why we call the url nifi-headless.nifi.svc.cluster.local , checking the list of nodes we validate that the new node is not part of the cluster, we delete the necessary files (users.xml, flow.xml.gz etc.), and we can start the NiFi node. By doing this, we avoid the situation where a new node start with all the other nodes downed, making the new node act as the referecing cluster, which will lead to lost everything :/

So for me everything looks fine, but you should be able to reach the UI ...