konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
125 stars 42 forks source link

Binding existing PVCs created previously by NiFiKop to a new cluster #277

Closed nitays closed 8 months ago

nitays commented 1 year ago

Is your feature request related to a problem?

I would like to create a NifiCluster which if I accidentally delete, i’ll be able to recreate it (with the same name and manifest) and have the PVCs retained and automatically bound to the NiFi nodes pods. I was able to successfully retain the PVCs (by changing the StorageClass setting), but I wasn’t able to connect the still-existent PVCs to a new NifiCluster.

Describe the solution you'd like to see

Having the operator first look for PVCs with certain labels\annotations and bind to them, before creating new PVCs for declared StorageConfigs.

Describe alternatives you've considered

I've considered provisioning the PVCs manually and binding them to the pods using externalVolumeConfig.

Additional context

That would be a really useful feature that would eliminate the urgent need for restrictive policies on deleting the NifiCluster on our environment, from the concern of having to migrate the PVCs content manually to new ones created by the new NifiCluster, especially on an environment in which data loss is critical.

mh013370 commented 9 months ago

We could add reclaimPolicy to nifikop's StorageConfig directly, supporting only Retain and Delete: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy

And then instead of strictly creating/deleting PVCs each time nifikop creates/kills a pod, it could perform lookups depending on the volume's reclaimPolicy: https://github.com/konpyutaika/nifikop/blob/master/pkg/resources/nifi/nifi.go#L150-L156

We already query for all of a pod's PVCs here: https://github.com/konpyutaika/nifikop/blob/master/pkg/resources/nifi/nifi.go#L164