search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
125 stars 42 forks source link

Make nifikop OpenShift compatible #254

Closed GitarPlayer closed 12 months ago

GitarPlayer commented 1 year ago
Q A
Bug fix? no yes
New feature? yes yes
API breaks? no yes
Deprecations? no yes
Related tickets mentioned in #250
License Apache 2.0

What's in this PR?

  1. I updated the kubebuilder RBAC annotations so the operator works on OpenShift.
  2. I added a new variable to install the helm operator while specifying the RunAsUser
  3. I added a nificlusters.nifi.konpyutaika.com sample for OpenShift

Why?

As it is NifiKop does not run on OpenShift without custom day 2 modifications

Additional context

I checked the install on OpenShift

# Tried on a clean AKS OpenShift cluster
oc version  
Client Version: 4.12.9
Kustomize Version: v4.5.7
Server Version: 4.10.54
Kubernetes Version: v1.23.12+8a6bfe4
# Create namespaces for Zookeeper and NiFi
oc create ns zookeeper
oc create ns nifi

# Install the CustomResourceDefinitions and cert-manager itself
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml

# Get UID range for the operator install from the nifi namespace
uid=$(kubectl get namespace nifi -o=jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' | sed 's/\/10000$//' | tr -d '[:space:]')

# Install Nifi operator using helm
helm install nifikop \
    nifikop \
    --namespace=nifi \
    --version 1.1.1 \
    --set image.tag=v1.1.1-release \
    --set resources.requests.memory=256Mi \
    --set resources.requests.cpu=250m \
    --set resources.limits.memory=256Mi \
    --set resources.limits.cpu=250m \
    --set namespaces={"nifi"} \
    --set runAsUser=$uid

# Get UID range for the Zookeeper operator from the zookeeper namespace
zookeper_uid=$(kubectl get namespace zookeeper -o=jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' | sed 's/\/10000$//' | tr -d '[:space:]')

# Get the default storage class for the cluster
sc=$(kubectl get storageclass -o=jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.metadata.name}{end}')

# Install Zookeeper using helm
helm install zookeeper bitnami/zookeeper \
    --set resources.requests.memory=256Mi \
    --set resources.requests.cpu=250m \
    --set resources.limits.memory=256Mi \
    --set resources.limits.cpu=250m \
    --set global.storageClass=$sc \
    --set networkPolicy.enabled=true \
    --set replicaCount=3 \
    --set containerSecurityContext.runAsUser=$zookeper_uid \
    --set podSecurityContext.fsGroup=$zookeper_uid \
    --namespace zookeeper

# Use the UID for the NiFi operator to set the fsGroup and runAsUser
sed -i "s/1000690000/$uid/g" config/samples/openshift

# Use the default storage class for the cluster to set the persistent volume claim
sed -i "s/standard/$sc/g" config/samples/openshift

# Apply the configuration for the NiFi operator
oc apply -f config/samples/openshift -n nifi

# Expose the NiFi service as a route
oc expose svc -n nifi simplenifi-headless

# Get the route for the NiFi service
route=$(kubectl get route simplenifi-headless -n nifi -o=jsonpath='{.spec.host}')

# Open the NiFi UI in Firefox using the route
firefox http://$route/nifi

Checklist

To Do

r65535 commented 1 year ago

Looks good to me!

It might be worth adding a page to the docs for running under openshift? Or adding a section to an existing page with the extra config required 😄

GitarPlayer commented 1 year ago

Looks good to me!

It might be worth adding a page to the docs for running under openshift? Or adding a section to an existing page with the extra config required 😄

Found the DOC src now, will do it. Done with the DOC and the changelog

mh013370 commented 1 year ago

I pulled this branch and built it just to make sure any kubebuilder-generated files get updated. I did find one that was missed (i.e. config/rbac/role.yaml). Can you please run make build locally and make sure that file gets committed?

Once that's done, i'm happy to merge. Thanks for the contribution :)

GitarPlayer commented 1 year ago

Sorry I was out of office for a week. My bad I added the make build changes.

mh013370 commented 1 year ago

@GitarPlayer : this PR needs its commits all signed and i can merge. Thanks!

gitguardian[bot] commented 1 year ago

⚠️ GitGuardian has uncovered 8 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
| GitGuardian id | Secret | Commit | Filename | | | -------------- | ------------------------- | ---------------- | --------------- | -------------------- | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Google OAuth2 Keys | 17f9d4a72aa8f7de7c1e231221083fca9dab8f30 | config/samples/tls_secured_nificluster.yaml | [View secret](https://github.com/konpyutaika/nifikop/commit/17f9d4a72aa8f7de7c1e231221083fca9dab8f30#diff-057714b9c25bd634fcec7978c1f8af99R31) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Google OAuth2 Keys | 684354e2869e13b6593f09bed801d02b99d30a0c | config/samples/tls_secured_nificluster.yaml | [View secret](https://github.com/konpyutaika/nifikop/commit/684354e2869e13b6593f09bed801d02b99d30a0c#diff-057714b9c25bd634fcec7978c1f8af99R31) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Google OAuth2 Keys | 06b28c3d3d93d7f0b039ef9d4636392733ce2cd3 | config/samples/tls_secured_nificluster.yaml | [View secret](https://github.com/konpyutaika/nifikop/commit/06b28c3d3d93d7f0b039ef9d4636392733ce2cd3#diff-057714b9c25bd634fcec7978c1f8af99R31) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Google OAuth2 Keys | 6d5c38dac0cb9518f74c908d3a91ee21060e72b5 | config/samples/tls_secured_nificluster.yaml | [View secret](https://github.com/konpyutaika/nifikop/commit/6d5c38dac0cb9518f74c908d3a91ee21060e72b5#diff-057714b9c25bd634fcec7978c1f8af99R31) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Google OAuth2 Keys | 7e83a6af5c6921e1e617a54405e5731352f70833 | config/samples/tls_secured_nificluster.yaml | [View secret](https://github.com/konpyutaika/nifikop/commit/7e83a6af5c6921e1e617a54405e5731352f70833#diff-057714b9c25bd634fcec7978c1f8af99R31) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Google OAuth2 Keys | f3b9aaf8a5495d45d67cc7bfc471e722d6b931a4 | config/samples/tls_secured_nificluster.yaml | [View secret](https://github.com/konpyutaika/nifikop/commit/f3b9aaf8a5495d45d67cc7bfc471e722d6b931a4#diff-057714b9c25bd634fcec7978c1f8af99R31) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Generic High Entropy Secret | 8d76820ebddd4a2afd2ce745288082ae628885c2 | site/website/docusaurus.config.js | [View secret](https://github.com/konpyutaika/nifikop/commit/8d76820ebddd4a2afd2ce745288082ae628885c2#diff-127a35840442280bcf41caf157c263e1R57) | | [-](https://dashboard.gitguardian.com/incidents/secrets) | Generic High Entropy Secret | 8f07ee25369e17410ad9f4417bd54292b73635dd | site/website/docusaurus.config.js | [View secret](https://github.com/konpyutaika/nifikop/commit/8f07ee25369e17410ad9f4417bd54292b73635dd#diff-127a35840442280bcf41caf157c263e1R59) |
🛠 Guidelines to remediate hardcoded secrets
1. Understand the implications of revoking this secret by investigating where it is used in your code. 2. Replace and store your secrets safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices. 3. Revoke and [rotate these secrets](https://docs.gitguardian.com/secrets-detection/detectors/specifics/google_oauth2#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). 4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. To avoid such incidents in the future consider - following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials - install [secret detection on pre-commit](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.

🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!

juldrixx commented 1 year ago

Why did this PR have 545 commits and old ones in it?

GitarPlayer commented 1 year ago

@GitarPlayer : this PR needs its commits all signed and i can merge. Thanks!

Can you point me out how to sign all my previous commits without making a git mess?

Why did this PR have 545 commits and old ones in it?

christian_bucheli@hotmail.com I tried signing my old commits but I ended up creating a mess. I am not sure what the correct procedure is to sign all my old commits.

juldrixx commented 1 year ago

@GitarPlayer : this PR needs its commits all signed and i can merge. Thanks!

Can you point me out how to sign all my previous commits without making a git mess?

Why did this PR have 545 commits and old ones in it?

christian_bucheli@hotmail.com I tried signing my old commits but I ended up creating a mess. I am not sure what the correct procedure is to sign all my old commits.

The issue is that you have a bunch of old commits the PR that are triggering an alert because of some secrets that aren't secrets.

GitarPlayer commented 1 year ago

@GitarPlayer : this PR needs its commits all signed and i can merge. Thanks!

Can you point me out how to sign all my previous commits without making a git mess?

Why did this PR have 545 commits and old ones in it?

christian_bucheli@hotmail.com I tried signing my old commits but I ended up creating a mess. I am not sure what the correct procedure is to sign all my old commits.

The issue is that you have a bunch of old commits the PR that are triggering an alert because of some secrets that aren't secrets.

This is related to the last three commits where I tried to sign all my commits. If I revert them / remove them from the history, this should be fixed. But do you know the easiest way to sign old commits done by me?

mh013370 commented 1 year ago

Hmm - i'm not sure how all of these commits got added, but I don't think we want to rewrite that much of the git history by merging. My recommendation would be to correct the commits on this branch or close this PR and raise another with only the signed commit(s) you want to contribute. Sorry for all the trouble for an otherwise simple contribution...

As far as signing old commits, i've found this helpful in the past: https://superuser.com/questions/397149/can-you-gpg-sign-old-commits

I'll often just squash all of my commits into 1 and make sure the one is signed. The easiest thing to do in this case might just be opening a new PR and cherry-pick your changes.

indiealexh commented 12 months ago

I'd love access to this compatibility feature this week, is there anything I can do to help this move forward?

juldrixx commented 12 months ago

I'd love access to this compatibility feature this week, is there anything I can do to help this move forward?

Fix the PR, as it is broken (500+ commits). And resolve the conflict.

GitarPlayer commented 12 months ago

I'd love access to this compatibility feature this week, is there anything I can do to help this move forward?

I will simply create a new fork and create a new PR and sign all my commits. This is easier than fixing this mess I've created.

GitarPlayer commented 12 months ago

I'd love access to this compatibility feature this week, is there anything I can do to help this move forward?

I opened a new PR that is clean and has its commits signed: https://github.com/konpyutaika/nifikop/pull/292