search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
125 stars 42 forks source link

Make nifikop OpenShift compatible #250

Closed GitarPlayer closed 9 months ago

GitarPlayer commented 1 year ago

Is your feature request related to a problem?

Currenlty nifikop does work with OpenShift due to missing RBAC permissions on finalizers: See here #https://github.com/konpyutaika/nifikop/issues/69

There are also some additional changes needed like for example the CRD NodeConfigGroups needs to be extended and more stuff. My question is if I implement these changes will they be merged or not?

Describe the solution you'd like to see

Nifikop runs on OpenShift and the getting started page explains how to get up and running with the specific OpenShift gotchas. Maybe nifikop even becomes a OpenShift certified operator but let's start small :-).

Describe alternatives you've considered

The Nifikop team has no interest in that (fair enough) and I consider a different nifi k8s project to make OpenShift friendly.

Additional context

I tried installing it on the latest OpenShift and it fails. I got 70% of the fixes to make it work on OpenShift and I wonder where to go with this.

mh013370 commented 1 year ago

This makes sense. I will try to look into this. Perhaps it's just correcting the RBAC yamls that get created by the Helm chart.

GitarPlayer commented 1 year ago

I would love to help, I tried installing the nifi operator on Red Hat OpenShift Local (https://developers.redhat.com/products/openshift-local/overview) with no success. When I ran make run it would return some error that the tls certificates were not found. But it was the first time for me to build a go operator with make so maybe I am also just a noop ;-). I followed this guide: https://konpyutaika.github.io/nifikop/docs/6_contributing/1_developer_guide I managed to install the operator on a test OpenShift cluster (I did have to edit the runAsUser and fsGroup of the nifikop pod so it would be in the namespace range by hand (see point 2).

  1. I added a lot of RBAC permissions to the nifikop role and unfortunately I did not keep track which permissions were missing exactly, because I just wanted to see if RBAC alone would resolve the problem.
  2. I tried deploying the simplenificluster.yaml but it would not deploy any pods because in OpenShift each project/namespace has a range of UID that is allowed to run pods and access shared storage (https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids). You can set runAsUser in the CRD so it is within that UID range of the namespace but I did not check if the fsGroup can be set as well in the NifiCluster resource. In any case if I can help you in some way please let me know. Also if you figure out how to install the operator on OpenShift local with make I would love to hear from you.
mh013370 commented 1 year ago

Thanks for that run-down! Were you able to actually get the operator installed successfully after these steps? Or are you still stuck?

Adding the /finalizers RBAC permission on all of the objects is a fairly easy thing for me to add, but i'm curious if you were able to resolve any of these issues.

GitarPlayer commented 1 year ago

I did manage to install the operator but I did not manage to install a nifi cluster yet through the operator. I will try again sometime this or next week, then I can maybe do a PR with the exact kubebuilder annotations for RBAC and maybe some OpenShift related DOC.

mh013370 commented 1 year ago

That'd be great! I'm also happy to help whip up these changes (kube builder annotations + helm chart Role mods). If you aren't able to get to it, sharing a working Role including all of the appropriate permissions needed ought to be good enough for me to go in and diff it all.

I'm admittedly very green when it comes to OpenShift (solely a Rancher-k8s user), but i should be able to help troubleshoot.

GitarPlayer commented 1 year ago

So I did some research and edited the RBAC kubebuilder annotations in my fork: https://github.com/konpyutaika/nifikop/compare/master...GitarPlayer:nifikop:master According to here: https://developers.redhat.com/blog/2020/09/11/5-tips-for-developing-kubernetes-operators-with-the-new-operator-sdk#tip_3__discovering_the_cluster_you_re_running_on

I set the runAsUser and fsGroup parameters in the CRD of config/samples/simplenificluster.yaml so it would run inside my nifi namespace. But it could not create the cluster because of filesystem permission issues:

oc logs -f  $(oc get po -l app=nifi -l nodeId=1 -n nifi --no-headers -o custom-columns=":metadata.name") -n nifi nifi
cp: cannot create regular file '/opt/nifi/nifi-current/conf/bootstrap-notification-services.xml': Permission denied  
cp: cannot create regular file '/opt/nifi/nifi-current/conf/bootstrap.conf': Permission denied  
cp: cannot create regular file '/opt/nifi/nifi-current/conf/logback.xml': Permission denied  
cp: cannot create regular file '/opt/nifi/nifi-current/conf/login-identity-providers.xml': Permission denied  
cp: cannot create regular file '/opt/nifi/nifi-current/conf/nifi.properties': Permission denied  
cp: cannot create regular file '/opt/nifi/nifi-current/conf/state-management.xml': Permission denied  
cp: cannot create regular file '/opt/nifi/nifi-current/conf/zookeeper.properties': Permission denied

This is from the nifi container and not from the zookeper container logs. This is not a nifikop operator problem because the image that is being used by the nifi container apache/nifi (https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/Dockerfile) has filepermission set that it only works for the user 1000. Sorry for that terrible formatting.

[ezra@U337659:/tmp/nifikop]$ podman run -d apache/nifi                                                                                                                        a1ae52de18dc191166ecf8595652c5a782d6295a7b360903bee95c11f645167e                                                                                                             
[ezra@U337659:/tmp/nifikop]$ podman exec -it -l 
nifi@a1ae52de18dc:/opt/nifi/nifi-current$ ll                                                                                                                                  total 360                                                                                                                                                                   drwxrwxr-x 1  
nifi nifi   4096 Apr 15 09:23 ./                                                                                                                               drwxr-xr-x 1 nifi   nifi   4096 Apr 10 20:05 ../                                                                                                                              drwxrwxr-x 1 nifi nifi     4096 Apr  3 21:28 bin/                                                                                                                             drwxrwxr-x 2 nifi nifi     4096 Apr 15 09:23 conf/                                                                                                                            drwxr-xr-x 2 nifi nifi     4096 Apr 10 20:05 content_repository/                                                                                                              drwxr-xr-x 2 nifi   nifi   4096 Apr 10 20:05 database_repository/                                                                                                             drwxrwxr-x 3   nifi nifi   4096 Apr  3 21:28 docs/                                                                                                                            drwxrwxr-x 2 nifi   nifi   4096 Apr  3 21:28 extensions/                                                                                                                      drwxr-xr-x 2 nifi   nifi   4096 Apr 10 20:05 flowfile_repository/                                                                                                             drwxrwx--- 6   nifi nifi  12288 Apr  3 21:28 lib/                                                                                                                             -rw-r--r-- 1 nifi   nifi 175405 Apr  3 21:28 LICENSE                                                                                                                          drwxr-xr-x 2 nifi   nifi   4096 Apr 15 09:23 logs/                                                                                                                            -rw-r--r-- 1 nifi nifi   110857 Apr  3 21:28 NOTICE                                                                                                                           drwxr-xr-x 2 nifi nifi     4096 Apr 10 20:05 provenance_repository/                                                                                                           -rw-r--r-- 1 nifi   nifi   4935 Apr  3 21:28 README                                                                                                                           drwxr-xr-x 2 nifi   nifi   4096 Apr 15 09:23 run/                                                                                                                             drwxr-xr-x 2 nifi nifi     4096 Apr 10 20:05 state/                                                                                                                           drwxr-xr-x 4 nifi nifi   4096 Apr 15 09:23 work/                                                                                                                             
nifi@a1ae52de18dc:/opt/nifi/nifi-current$ id                                                                                                                                 
uid=1000(nifi) gid=1000(nifi) groups=1000(nifi)

So there are two options: 1. reference a custom built container image
2. ask the apache/nifi team to adjust filesystem permissions
I already mailed them regarding this matter but I am not sure they will do it since it is quite particular to OpenShift to have guid 0 rwX permissions on the relevant files. They would profit greatly if they would use an image like ubi9/openjdk-11-runtime as a base since this is regularly patched and scanned for vulnerabilities and minimal in size. I am not 100% certain that these RBAC fixes make it work entirely, I would first have to build a OpenShift compatible nifi image and verify if the rest works smoothly as well. I will report this as soon as I know more.

GitarPlayer commented 1 year ago

I built all nifikop compatible images here: https://quay.io/repository/rhnclimocha/nifi?tab=tags. They are based on ubi9/openjdk-11-runtime and are built on a CRON schedule to get security patches for the OS. As soon I have time I will try if nifikop works with the fixed RBAC and the custom docker image.

GitarPlayer commented 1 year ago

I managed to get the simplenifi cluster running on OpenShift by using my own images quay.io/rhnclimocha/nifi:1.7.0 and setting the fsGroup and runAsUser settings particular to my nifi namespace. I will do a PR request that outlines all the steps for it to work on OpenShift. Could you merge the updated RBAC kubebuilder annotations?

mh013370 commented 1 year ago

Apologies! I was out last week at KubeCon. I'll jump back into this :)

Thanks for your patience.