search

B23admin / nifi-stateless-operator

An Operator for scheduling and executing NiFi Flows as Jobs on Kubernetes
Apache License 2.0
53 stars 12 forks source link

Failed to list *v1alpha1.NiFiFn: nififns.nififn.nifi-stateless.b23.io is forbidden #8

Closed nagwanidheeraj closed 5 years ago

nagwanidheeraj commented 5 years ago

Followed the readme.md instructions and configured nifi-stateless-operator on an AWS EKS Cluster.

When I see the logs for the manager, I see this:

kubectl -n nifi-stateless-operator-system logs nifi-stateless-operator-controller-manager-64b64fb5d6-9rz9k manager


2019-09-19T12:27:48.538Z        INFO    controller-runtime.controller   Starting EventSource    {"controller": "nififn", "source": "kind source: /, Kind="}
2019-09-19T12:27:48.538Z        INFO    controller-runtime.controller   Starting EventSource    {"controller": "nififn", "source": "kind source: /, Kind="}
2019-09-19T12:27:48.538Z        INFO    setup   starting manager
2019-09-19T12:27:48.735Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
E0919 12:27:48.835759       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: Failed to list *v1alpha1.NiFiFn: nififns.nififn.nifi-stateless.b23.io is forbidden: User "system:serviceaccount:nifi-stateless-operator-system:default" cannot list resource "nififns" in API group "nififn.nifi-stateless.b23.io" at the cluster scope
2019-09-19T12:27:48.841Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"ConfigMap","namespace":"nifi-stateless-operator-system","name":"controller-leader-election-helper","uid":"e43aec29-dad8-11e9-912e-0a4e6c6c1e40","apiVersion":"v1","resourceVersion":"111130"}, "reason": "LeaderElection", "message": "nifi-stateless-operator-controller-manager-64b64fb5d6-9rz9k_e40d2f9c-dad8-11e9-93b3-3ad7f9eb4e9f became leader"}
E0919 12:27:49.841317       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: Failed to list *v1alpha1.NiFiFn: nififns.nififn.nifi-stateless.b23.io is forbidden: User "system:serviceaccount:nifi-stateless-operator-system:default" cannot list resource "nififns" in API group "nififn.nifi-stateless.b23.io" at the cluster scope
E0919 12:27:50.842550       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: Failed to list *v1alpha1.NiFiFn: nififns.nififn.nifi-stateless.b23.io is forbidden: User "system:serviceaccount:nifi-stateless-operator-system:default" cannot list resource "nififns" in API group "nififn.nifi-stateless.b23.io" at the cluster scope
E0919 12:27:51.843717       1 reflector.go:126] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: Failed to list *v1alpha1.NiFiFn: nififns.nififn.nifi-stateless.b23.io is forbidden: User "system:serviceaccount:nifi-stateless-operator-system:default" cannot list resource "nififns" in API group "nififn.nifi-stateless.b23.io" at the cluster scope```
dbkegley commented 5 years ago

Thanks for taking an interest in this project! Unfortunately, I won't have time to look into this bug in the near term but I would be happy to review a PR or any additional info if you'd like to dig into it. The RBAC roles are generated from the kubebuilder annotations here: https://github.com/b23llc/nifi-stateless-operator/blob/master/controllers/nififn_controller.go#L17-L20

You can regenerate the roles and rolebindings with the command: make manifests

nagwanidheeraj commented 5 years ago

@dbkegley - Thank you for taking the time to respond to the issue, I was able to fix the issue based on your response.

Please review the commit linked above, if you are ok with the changes, I will raise a PR.

PS: I also have some changes to make flowFiles optional and support pulling images from a private repository by way of image pull secrets. I will push those in a separate commit.

dbkegley commented 5 years ago

@nagwanidheeraj That looks good to me, thanks for looking into this! I'll test the PR when it's posted

Out of curiosity, what's the motivation behind making flowfiles optional? I'm not opposed, just wondering what your use case is and how you plan to enqueue flowfiles if none are provided at runtime

nagwanidheeraj commented 5 years ago

@dbkegley - Raised the PR

Out of curiosity, what's the motivation behind making flowfiles optional? I'm not opposed, just wondering what your use case is and how you plan to enqueue flowfiles if none are provided at runtime

Most of the use cases I have are where the data is coming from external sources like s3. We start with the ListS3 bucket processor and go from there. This is true for a majority of our use cases with s3 being the main source, and some are also based on GCS/Kafka/SQS and such. No NiFi flow that we have requires an input file, yet.

Is there a way we can chat further about the changes I'm planning over an email? My email address is listed on my profile and would love to pick your brain.

dbkegley commented 5 years ago

Absolutely, feel free to shoot me an email at kegs@b23.io

One thing that I want to mention here in case others are interested is that there is currently a requirement in nifi-stateless imposed here and here which will fail initialization if there is not exactly one input port at the root level of the flow. I have some initial thoughts on this and a very similar use case to yours but I haven't proposed them in the nifi developer list yet. I was planning to wait until after the first official release of nifi-stateless in nifi-1.10

nagwanidheeraj commented 5 years ago

Those requirements mean:

I was able to run a flow on EKS that has no input ports and no flow files specified in the NiFiFn yaml. Flow executed successfully and was able to copy a file from one folder to another in a s3 bucket.

nagwanidheeraj commented 5 years ago

You can build the enhancements branch in my fork and try it out yourself. It's been tested to see if it works.

dbkegley commented 5 years ago

Ah, you're right. I hadn't actually tested it, that's just what I remembered from looking through the nifi-stateless code previously. In that case I agree that the flow_files attribute should be optional for the operator

supriyaK56 commented 4 years ago

Those requirements mean:

  • there can only be one input port
  • if there is no input port, flow file cannot be enqueued

I was able to run a flow on EKS that has no input ports and no flow files specified in the NiFiFn yaml. Flow executed successfully and was able to copy a file from one folder to another in a s3 bucket.

@nagwanidheeraj which image you used for this? I want to run flow without input ports