search

saidsef / k8s-nifi-cluster

Apache NiFi cluster running in Kubernetes
Apache License 2.0
49 stars 21 forks source link

Node adress gets 0.0.0.0:8080 and not pod ip #15

Closed ben749l closed 1 year ago

ben749l commented 1 year ago

@saidsef Hello, thank you for your work. I tried to deploy Nifi Cluster in Kubernetes (Statefulset and also tried Deployment in kind value) and use same deployment but cluster node adress doesn't get the ip pod but 0.0.0.0:8080. I am able to have several nodes but they have same adress and I have this error in NiFi : Node 0.0.0.0:8080 is unable to fulfill this request due to: Transaction XXX is already in progress. Do you know where can it come from ?

github-actions[bot] commented 1 year ago

👋 Thanks for Reporting!

saidsef commented 1 year ago

@ben749l Happy New Year!

This sounds very familiar, and if I recall correctly it depends on the version and/or K8s cluster is configured.

Anyways, I've created PR #16 fix, essentially NIFI_WEB_HTTP_HOST should be the pod address - I've noticed this behaviour depends on the cluster. Now NIFI_WEB_HTTP_HOST will be set per pod via status.podIP, have tested this and it seems to resolve the issue.

Once merged, it should updated release/tag v2023.01

ben749l commented 1 year ago

Happy new year ! Thank you very much for your answer, it solved my problem :) Is there a prefered value for kind ? (deployment or statefulset) Maybe not the place to ask several questions after a solved problem

saidsef commented 1 year ago

Glad that helped.

Since kind is a lightweight Kubernetes cluster you'll just need to explicitly declare endpoints/services addresses. Also, depending on your workload you might want to give it a bit more cpu and memory, the current resources values are a starting point.

ben749l commented 1 year ago

Thank you ! How did you get the value auth: YWRtaW46JGFwcjEkSDY1dnBkTU8kMXAxOGMxN3BuZVFUT2ZjVC9TZkZzMQo= for the secret ?

saidsef commented 1 year ago

That is bse64 encoded of http basic auth using htpasswd, the username and password are admin/admin - this is access via ingress, please change that asap if it is public facing!

ben749l commented 1 year ago

If I have to change it, do I have to use port 8443 to use HTTPS ?

saidsef commented 1 year ago

The short answer is no. To use 8443/HTTPS you'll need to load a cert in the container.

The way I've setup the service is HTTPS/SSL termination at the LoadBalancer.

ben749l commented 1 year ago

Thanks again ! Did the way you set up allows to login and enter password ? I am trying it with Traefik but I guess I have to change port to 8443 and do something else. Nifi doesn't ask me to authenticate, UI says "Anonymous user" and when I click on "LOG IN" button it opens a pop up saying:

"Unable to check Access Status User authentication/authorization is only supported when running over HTTPS."

saidsef commented 1 year ago

The username/password challenge is via ingress controller which is nginx.

https://github.com/saidsef/k8s-nifi-cluster/blob/1ba029d124820a219003aaba8bfc7bbbfe5176b1/deployment/ingress.yml#L11-L13

It will still say "Anonymous user", NiFi has it's own authentication process and you'll need to configure that. Sounds like you are using Traefik for ingress, unfortunately that is out of scope.

Also, port 8443 has not been exposed due to cert requirements - I might do that at a later date, here are a list of exposed ports:

https://github.com/saidsef/k8s-nifi-cluster/blob/1ba029d124820a219003aaba8bfc7bbbfe5176b1/deployment/nifi-deployment.yml#L67-L73

ben749l commented 1 year ago

Thank you again for all the answers !