search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
136 stars 45 forks source link

NiFi clustering without ZooKeeper #448

Closed malthe closed 5 days ago

malthe commented 2 months ago

Is your feature request related to a problem?

NiFi 2.0.0 introduces new capabilities leveraging Kubernetes ConfigMaps and Leases to support clustering without the need for ZooKeeper.

We should leverage this capability to simplify the deployment.

Describe the solution you'd like to see

See https://exceptionfactory.com/posts/2024/08/10/bringing-kubernetes-clustering-to-apache-nifi/ for additional background and motivation.

Describe alternatives you've considered

No response

Additional context

No response

juldrixx commented 2 months ago

Hello,

We had started work on implementing the feature in this PR but we were missing a feature in NiFi for which we had requested validation of the implementation (https://issues.apache.org/jira/browse/NIFI-12590) so we were waiting for the feature to be released. But since then, we haven't necessarily had the time to go back over it.

But yes, we want to implement it. And it will maybe lead to a new CRD version for the NifiCluster.

yakovasavr commented 6 days ago

Hello team,

It seems issue has beed resolved on Apache side. Could you please return back to the feature? It is quite important for product future to be able to provide a solution without zookeeper.

juldrixx commented 6 days ago

Hello team,

It seems issue has beed resolved on Apache side. Could you please return back to the feature? It is quite important for product future to be able to provide a solution without zookeeper.

The feature is ready with its PR (https://github.com/konpyutaika/nifikop/pull/450) It will be merged tomorow if there is no more review. And a new version will be released with it.