search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
131 stars 45 forks source link

Manage autoscale 0 and no node cluster #300

Closed juldrixx closed 1 year ago

juldrixx commented 1 year ago
Q A
Bug fix? yes
New feature? yes
API breaks? no
Deprecations? no
Related tickets -
License Apache 2.0

What's in this PR?

Feature allowing NifiClusternot to define the nodes in the spec or to make them an empty list. Fix to handle empty CreationTime in node states.

Why?

In the case of a cluster managed by autoscaling, we want to be able to make autoscaling go all the way down to 0. But in the case where autoscaling manages all nodes, we need to allow a cluster to be no node at all.

Additional context

When deploying clusters using chart helm, we don't want the replicas field of the NifiNodeGroupAutoscaler and the nodes field of the NifiCluster to be defined, to avoid rewriting them in the event of a chart update. To do this, these fields must be optional and therefore completely managed in this case (= no node).

Checklist

mh013370 commented 1 year ago

Sorry, i meant to review this. I mentioned this as a limitation when i contributed this feature:

https://github.com/konpyutaika/nifikop/pull/89

It's not possible to deploy a NifiCluster with only autoscaled node groups. The NifiCluster CRD requires that you specify at least one node in the spec.nodes list. Do we want to support this? If so, we may need to adjust the cluster initialization logic in the NifiCluster controller.

So it's nice to see this added. Is there an example of a minimum NiFiCluster + NifiNodeGroupAutoscaler configuration in the nifikop docs?

juldrixx commented 1 year ago

I haven't really changed the documentation, I've just corrected a few things because it's still true. But I guess the minimum configuration will be a NifiCluster without node and a NifiNodeGroupAutoscaler without replicas fully controlled by an HPA.