Need for editing /etc/hosts file?

[PrashanthVenkatesan]

Hi, Actually by creating headless service , interpod communication can happen with FQDN ( podname.svcname.namespace.svc.cluster.local ) . There is no need of mapping it to hostname ( in /etc/hosts file ), we can give FQDN as hostname in nifi.properties. This will resolve your scalability challenge as well, right?

[AlexsJones]

The problem I've found is that zookeeper keeps a record that the nifi master is (for example) nifi-0 (That name comes from the $HOSTNAME of the current STS pod) however what is discoverable is nifi-0.nifi.nifi.svc.cluster.local that's the only reason I map it. Which then lets nifi resolve it's cousin node.

See what I mean? Happy for suggestions though...

Also, This issue is way more pertinent when using SSL mode which requires the in the niif.properties file, all the nodes to be defined pre-runtime It seems to me the nifi project was designed to restart nodes when there is a new node being added to the cluster in the properties or users.xml and authorizers.xml

[AlexsJones]

If we could override the hostname to match the podname.svcname.namespace.svc.cluster.local I agree this could work - it would mean ZK keeps that fully resolvable record.

[cmccaw]

The hostname can match podname.svcname.namespace.svc.cluster.local using the templating language when creating a helm chart

[AlexsJones]

I am going to look into this now thanks @cmccaw

[cmccaw]

The problem I have hostname is that the hostname needs to match the Host header . Otherwise, Nifi fails (https://community.hortonworks.com/questions/164783/nifi-15-system-error-invalid-host-header.html) How did you get around this? I had to create an Nginx Ingress and forced the Host to nifi-0 This doesn't work under a cluster tho since the service and select any of the pods. Not just nifi-0

[PrashanthVenkatesan]

For ssl, we can create certs with wildcard host like *.svcname.namespace.svc.cluster.local . then you can change your nifi docker scripts to populate the output of hostname -f in all the places where nifi requires hostname in nifi.properties. This will make your pod communicated among each other via FQDN . @cmccaw do you have any ref for updating hostname via helm

[PrashanthVenkatesan]

@cmccaw Looks like your issue is resolved in nifi-1.6.0 ( recently released )

[cmccaw]

In my docker file I have a script that replaces the hostname in nifi.properties with the environment variable HOSTNAME My helm then looks like this which can communicate to each other via FQDN

env
  - name: POD_NAME
    valueFrom:
      fieldRef:
      fieldPath: metadata.name
   - name: HOSTNAME
       value: "$(POD_NAME).{{$fullName}}.default.svc.cluster.local"

[AlexsJones]

@cmccaw :pray:

Does this mean you're forced to bake the nifi folder into the dockerfile or do you dynamic mount it?

[cmccaw]

@AlexsJones My dockerfile moves the whole NiFi (1.6.0 in my case) into the container. I mount the NiFi repositories though. I base from FROM apache/nifi:1.6.0 and then my dockerfile has some extra stuff in it

[juv]

@PrashanthVenkatesan @cmccaw mind sharing your kubernetes configuration files?

[svrc]

another idea in kubernetes 1.9+ is to use dnsConfig, which would allow you to add the service domain (nifi.nifi.svc.cluster.local) as a search domain for the pods, thereby $HOSTNAME will work unqualified.

this requires a feature gate on your k8s cluster.

  spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - nifi
              topologyKey: "kubernetes.io/hostname"
      dnsConfig:
        searches:
          - nifi.nifi.svc.cluster.local
      containers:
        - name: nifi
          image: xemuliam/nifi:1.6.0
          command: ["/bin/sh","-c"]
          args: ["cd /opt/nifi && ./start_nifi.sh"]

more info: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

[kfox1111]

you might be able to just set HOSTNAME variable to the fqdn version too. It already seems to be set to the right value. root@web-2:/# echo $HOSTNAME web-2 root@web-2:/# HOSTNAME=$(hostname -f) root@web-2:/# echo $HOSTNAME web-2.nginx.default.svc.cluster.local

[mcneary70]

I'm using Deployments instead of STS but was having similar problems. My kludge was to set hostNetwork: true and mount a custom resolv.conf via configMap. It isn't pretty, but it did work for me for now.

[nicknezis]

Would it be possible to add and remove hosts using the NIFI Rest api from a k8s operator? https://nifi.apache.org/docs/nifi-docs/rest-api/index.html /controller/cluster/nodes/{id}

[AlexsJones]

another idea in kubernetes 1.9+ is to use dnsConfig, which would allow you to add the service domain (nifi.nifi.svc.cluster.local) as a search domain for the pods, thereby $HOSTNAME will work unqualified.

this requires a feature gate on your k8s cluster.

  spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - nifi
              topologyKey: "kubernetes.io/hostname"
      dnsConfig:
        searches:
          - nifi.nifi.svc.cluster.local
      containers:
        - name: nifi
          image: xemuliam/nifi:1.6.0
          command: ["/bin/sh","-c"]
          args: ["cd /opt/nifi && ./start_nifi.sh"]

more info: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

This is a really good idea, I'll give a whirl ASAP.

For now I've got rid of the gross shell script and at least put it in the STS directly

[AlexsJones]

Great thread and thanks for the help everyone, now using externalServices and it works much cleaner!