Support for 1.14.0

[riccardo-salamanna]

Feature Request

Testing of nifikop with version 1.14.0 have been unsuccessful.

Describe the solution you'd like to see It would be nice since to have that support, since the release fixes some important bugs for us

Many thanks

[mh013370]

I was able to get it working. The nifi logs inform you what needs to be added to get it to work.

Can you clarify specifically what isn't working?

[iordaniordanov]

I got it working too, the only change I had to make was to explicitly provide nifi.sensitive.props.key in overrideConfigs. It is working with 1.15.0 too

[riccardo-salamanna]

nifi.sensitive.props.key

would you mind shring the manifest for the cluster with your settings? i suspect mi issues comes from using the certificate manager... thanks

[iordaniordanov]

Oh, yeah, I'm not using cert-manager

[iordaniordanov]

You can see my manifest here - https://github.com/Orange-OpenSource/nifikop/issues/139 , as I said I only added nifi.sensitive.props.key

[mh013370]

nifi.sensitive.props.key

would you mind shring the manifest for the cluster with your settings? i suspect mi issues comes from using the certificate manager... thanks

I did just recently discover #153

But otherwise, it should work just fine. I've cert-manager enabled (deployed version v1.5.4) in my setup without issues.

Are there errors in the nifikop logs or the nifi-app logs?

[riccardo-salamanna]

nifi.sensitive.props.key

would you mind shring the manifest for the cluster with your settings? i suspect mi issues comes from using the certificate manager... thanks

I did just recently discover #153

But otherwise, it should work just fine. I've cert-manager enabled (deployed version v1.5.4) in my setup without issues.

Are there errors in the nifikop logs or the nifi-app logs?

It is hard to debug since Nifikop fails to start the pods and kills them before i can debug... and all i did was change the version on my working setup

[riccardo-salamanna]

I got it working too, the only change I had to make was to explicitly provide nifi.sensitive.props.key in overrideConfigs. It is working with 1.15.0 too

I just tested with the simplecluster.yaml just changing the versions and it doesn't work for me. Thanks for all your support :)

[erdrix]

For the nifi.sensitive.props.key we are balanced between letting people configure it using the nifiProperties. overrideSecretConfig or explicitly set a secretconfigreference for this nifi property.

We have not tested the version 1.14 and 1.15 so we don't explicitly support these two version (We have some rework to replace the unsecured mode which has changed and require new configurations : username and password).

[erdrix]

We are working on the integration with NiFi 1.14 and NiFi 1.15, it seems after some tests that everything works perfectly with these new versions (if you add property nifi.sensitive.props.key in the nifi.properties) in case you are already on a secured cluster.

For unsecure clusters, I don't touch all the things that it involves, but if there isn't a detailed clear process somewhere, I assume that the operator won't support the upgrade.

[riccardo-salamanna]

We are working on the integration with NiFi 1.14 and NiFi 1.15, it seems after some tests that everything works perfectly with these new versions (if you add property nifi.sensitive.props.key in the nifi.properties) in case you are already on a secured cluster.

For unsecure clusters, I don't touch all the things that it involves, but if there isn't a detailed clear process somewhere, I assume that the operator won't support the upgrade.

Thanks @erdrix - I will try ASAP and report how it goes

[yossisht9876]

@erdrix my nifi setup include cert manger version 1.2 , after adding the key "nifi.sensitive.props.key" the pods keep restarting in a loop and i cant debug it. do you test it with cert manger enabled ? any ideas ? thanks

[yossisht9876]

deleting the certs related to nifi forced the cert manger to create new certs and now its working

[erdrix]

We are working on the integration with NiFi 1.14 and NiFi 1.15, it seems after some tests that everything works perfectly with these new versions (if you add property nifi.sensitive.props.key in the nifi.properties) in case you are already on a secured cluster. For unsecure clusters, I don't touch all the things that it involves, but if there isn't a detailed clear process somewhere, I assume that the operator won't support the upgrade.

Thanks @erdrix - I will try ASAP and report how it goes

After some additional test we faced to some issues during the upgrade, the following actions should works to migrate from 1.12.X or 1.13.X to 1.14+ :

1. Edit your NiFiCluster resource by adding an init container :

...
spec
  ...
  initContainers:
  ...
  - command:
    - sh
    - -c
    - '/opt/nifi/nifi-current/bin/nifi.sh set-sensitive-properties-key <your encryption password>'
    image: apache/nifi:1.14.0
    imagePullPolicy: IfNotPresent
    name: upgrade-nifi
    volumeMounts:
    - mountPath: /opt/nifi/data
      name: data
    - mountPath: /opt/nifi/nifi-current/conf
      name: conf
  ...

2. Add your sensitive key in your nifi.properties (using one of the three possibility, it is highly recommended to use a secret) : nifi.sensitive.props.key=<your encryption password> .

3. Force the deletion of all your NiFi node's pods, in a way that they will start in the same time.

4. Once all the pods of your cluster are up, edit once again your NiFiCluster resource, by removing the added initContainer and upgrading your NiFi version :

spec
  ...
  clusterImage: apache/nifi:1.15.1
  ...

And your cluster should be running :)