search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
125 stars 42 forks source link

Is NiFi KOP increases load on k8s cluster while watching/monitoring crds #260

Closed narayanbhawar10 closed 1 year ago

narayanbhawar10 commented 1 year ago

Type of question

Best Practices

Support question

Is NiFi KOP continuously hits to k8s When NiFi KOP watches crds and monitors as it is shown in thread dump metrics of NiFi KOP . I just want to understand if it increases load on k8s cluster.

NiFiKop version

No response

Golang version

No response

Kubernetes version

No response

NiFi version

No response

r65535 commented 1 year ago

Yes, NiFiKop queries the kubernetes API server to make sure no work needs to be done, before requeuing and trying again. This is a standard way of implementing an operator, so the more you have installed, the more API calls there are being made. This load should be negligible, unless you're right on the edge of a control plane node having resourcing issues.

If everything is working correctly, requeuing is every 15 seconds per-CRD: https://github.com/konpyutaika/nifikop/blob/master/pkg/common/common.go#L74-L89. This can be tweaked by adding the ENV vars found in my link to the operator pod.

If resources need changing or there are errors, NiFiKop will requeue more aggressively and exponentially back off.

narayanbhawar10 commented 1 year ago

Understood Thank you for your response.

mh013370 commented 1 year ago

I want to add one additional detail. Normally, an operator will be provided with an event any time a CRD the operator is watching changes. So most of the time it's not necessary to continuously query the k8s API. With nifikop however, it also ensures that the dataflow deployed to a NiFi canvas doesn't get changed by users. More precisely, nifikop will revert any change a user makes to a deployed NifiDataflow where its syncMode is always. This canvas change wouldn't be reflected in any CRD, so this is the primary reason nifikop continuously queries the k8s API.

All of that said, the rate that each nifikop controller queries the k8s API is configurable. By default, it's 15 seconds, which is generally low volume:

https://github.com/konpyutaika/nifikop/blob/master/pkg/common/common.go#L81-L87