Additional info in documentation

[tide34]

Requirement - what kind of business use case are you trying to solve?

Documentation link.
Add more information about role with name jaeger-operator in case when deploy jaeger in not cluster-wide mode.

The problem with i faced is misunderstanding documention.
I want to deploy application in not cluster wide mode (run jaeger-operator in a namespace (like, observability) and manage CR in another namespace (like, myproject)).
I have changed clusterRole -> role, clusterRoleBinding (except jaeger-operator-metrics-reader due to usage nonResourceURLs inside) -> roleBinding in this manifest, add annotations in jaeger-operatot deployment manifest:

annotations:
olm.targetNamespaces: myproject

to set WATCH_NAMESPACE environment variable and deploy jaeger-operator in observability namespace. Create RoleBinging like at the end of paragraf here and see restarts of jaeger-operator in observability namespace due to next errors:

time="2022-05-19T19:31:37Z" level=error msg="error getting a list of deployments to analyze in namespace" error="deployments.apps is forbidden: User \"system:serviceaccount:observability:jaeger-operator\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"myproject\": RBAC: role.rbac.authorization.k8s.io \"jaeger-operator\" not found" namespace=myproject time="2022-05-19T19:31:37Z" level=error msg="error getting a list of existing jaeger instances in namespace" error="jaegers.jaegertracing.io is forbidden: User \"system:serviceaccount:observability:jaeger-operator\" cannot list resource \"jaegers\" in API group \"jaegertracing.io\" in the namespace \"myproject\": RBAC: role.rbac.authorization.k8s.io \"jaeger-operator\" not found" namespace=myproject

Next strange think for me is log line:

time="2022-05-19T19:31:31Z" level=info msg="Consider running the operator in a cluster-wide scope for extra features"

Why jaeger-operator think that he works in cluster-wide spoce when i configure to watch only myproject namespace and create role and roleBinding resources?

Proposal - what do you suggest to solve the problem or improve the existing situation?

Add more information about role manifests in namespaces when use jaeger-operator in not cluster-wide mode (watch only some namespaces)

[yurishkuro]

would you like to propose specific changes via a PR? https://github.com/jaegertracing/documentation/tree/main/content/docs/next-release

[tide34]

no, i would ask for someone who knows how to run and up jaeger-operator in not cluster-wide mode to complete documentation.
I have already spent three days to run jaeger operator locally and still haven't done it due to documentation misunderstanding

[tide34]

I also try creat role in myproject namespace which solve log errors like in this:

time="2022-05-19T19:31:37Z" level=error msg="error getting a list of deployments to analyze in namespace" error="deployments.apps is forbidden: User \"system:serviceaccount:observability:jaeger-operator\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"myproject\": RBAC: role.rbac.authorization.k8s.io \"jaeger-operator\" not found" namespace=myproject time="2022-05-19T19:31:37Z" level=error msg="error getting a list of existing jaeger instances in namespace" error="jaegers.jaegertracing.io is forbidden: User \"system:serviceaccount:observability:jaeger-operator\" cannot list resource \"jaegers\" in API group \"jaegertracing.io\" in the namespace \"myproject\": RBAC: role.rbac.authorization.k8s.io \"jaeger-operator\" not found" namespace=myproject

but after that new errors message like this appears:

W0519 19:19:33.212583 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.0/tools/cache/reflector.go:167: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:observability:jaeger-operator" cannot list resource "namespaces" in API group "" at the cluster scope

i have checked that service account of jaeger-operator bind to role (not clusterRole due to i wanna watch to only one namespace) and the role has right permition to list namespaces in group "".

[tide34]

I think it's impossible to start jaeger-operator in not cluster-wide mode when follow steps in documention.

Could your team can test it and give response about statement above?

[tide34]

@yurishkuro hello. Will anyone help solve it?

[frzifus]

@rubenvp8510 Do you have experience with it?

[tide34]

no one help

[rubenvp8510]

Let me try follow your steps, and will back to you with more information, we might need to update the documentation. Thanks for reporting this!

[rubenvp8510]

Could you attach your manifest yaml file?

[rubenvp8510]

I tried to install it on restricted mode (only watching observability namespace) and it works for me.

A couple of points you can check. See if the Roles and RoleBindings are created in the respective namespace. I was able to reproduce the issue with the following command:

kubectl apply -f manifest.yaml

But if I do this: kubectl apply -f manifest.yaml -n observability

The error of the namespace goes away, and the operator start successfully.

My manifest: jaeger-operator.txt

[tide34]

Steps (use minikube):
1) install cert-manager v1.6.3 (follow this guide)
2) download jaeger-operator manifest (from documentation)
3) as documentation told if i want to install jaeger-operator in non cluster wide mode (watch jaeger CR in specific namespaces ONLY, in my case for example namespace app-profile) i need to change the ClusterRole and ClusterBindingRole of the operator manifest to Role and RoleBinding, also set the WATCH_NAMESPACE environmebt variable on the jaeger operator Deployment. Have done it. Got next manifest

kubectl apply -f jaeger-operator.yaml -n observability

4) create this manifest in namespace app-profile (documentation link. I think that in documentation absent manifest of role jaeger-operator in namespace to watch jaeger CR):

kubectl create ns app-profile
kubectl apply -f additional.yaml

---

in log still see line:
time="2022-05-29T07:17:46Z" level=info msg="Consider running the operator in a cluster-wide scope for extra features"
Pod of jaeger-operator restarts

[tide34]

@rubenvp8510 can you try to install jaeger-operator in mode, when look for jaeger CR in not observability namespace

[tide34]

apply manifest from your post above and see in log that jaeger-operator run in not cluster wide mode.

pod of jaeger operator also has restarts

[rubenvp8510]

The log line: "Consider running the operator in a cluster-wide scope for extra features

Is normal, it is only indicating that some features are not present on the namespace scoped operator. I'll check your manifest and see if I can reproduce your error.

[tide34]

About log ok, i undestand.
Wait for your answer
Check your manifest. Pod of jaeger operator restarting. I tried to deploy operator used your manifest

[rubenvp8510]

I see the problem, Indeed there is a regression here!

The namespace controller should not be enabled when the operator is not running in cluster wide mode. I'm sending a PR to fix this. hope we can do a minor release this week.

Thanks for reporting.

[tide34]

Great, we have found issue. Yhanks for your work. I will look forward to closing PR.

what about that question?

4. create this manifest in namespace app-profile (documentation link. I think that in documentation absent manifest of role jaeger-operator in namespace to watch jaeger CR)
   Can you add in documentaion manifest of jaeger-operator role? - i think it will help a lot more people to deploy application quickly without misunderstanding. Now this manifest is absent

[rubenvp8510]

About point 4, wondering why do you need that? I think the reason is that you are trying to apply the manifest in the observability namespace: kubectl apply -f jaeger-operator.yaml -n observability instead you should apply it on your own namespace: kubectl apply -f jaeger-operator.yaml -n app-profile. (But you need to change all references to observability on the manifest)

And may be we need to clarify that on the documentation. Make sense ?

[tide34]

i want to deploy jaeger-operator in observability manifest and watch for jaeger CR in app-profile namespace. As documentaion said:

It is possible to have the Jaeger Operator running in a given namespace (like, observability) and manage Jaeger resources in another (like, myproject).

I want to use this mode. After reading documentation i think that only need to create that role in app-profile namespace and nothing more. I got confused what role with name jaeger-operator (role with that name ansebt in this manifest too) should be deployed because role it is namespaced resource.

As i undestand documentation, i have described stepts in guide above what i need to do to deploy jaeger with described scheme of work.

[rubenvp8510]

Aaa sorry I misunderstood a little bit your scenario, yeah in that case we might want to add a note to the documentation. I'll create the issue and the PR tomorrow :)

[tide34]

I will close the issue when check that all work correctly and pod of jaeger-operator don't restarts.

Thanks a lot for help

[tide34]

@rubenvp8510 hello. Please, don't forget create PR

[rubenvp8510]

@tide34 I created the first PR: https://github.com/jaegertracing/jaeger-operator/pull/1914 here, I will create a new one on the docu repo to improve it for the use case you mention.

[tide34]

@rubenvp8510 Hello. When are you planning to create release with fix?

[tide34]

@rubenvp8510 ?

[rubenvp8510]

Sorry I haven't had time to see this, I'm evaluating if we are going to do with a minor release, or do 1.25. because jaeger 1.25 is out.

[tide34]

Maybe you have mistake in previous message and wrote about version 1.35?
Can you create a patch release in 1.34.x? I am really looking forward it)

[tide34]

Have you decided the release date of 1.35?

[frzifus]

Hi @tide34, i will work on 1.35 next week.

[tide34]

Hi @frzifus. Please give some info about release date.

[frzifus]

probably tomorrow https://github.com/jaegertracing/jaeger-operator/pull/1949

[tide34]

Hi, @rubenvp8510. Have you already created issue for documention impruvment? I can't find issue in respository with docs

[tide34]

@rubenvp8510 @frzifus thank a lot for new release. My problem has solved
@rubenvp8510 i still confused about documentation. I'm seeing that documentation here doesn't tell anythink at all about what Role i need to create in namespace where jaeger resources wanted to traced by jaeger operator.

Could you help with it, please

[eakurdyukov]

+1 docs looks confusing for me too. Need more examples maybe

[sereneshikari]

Yes, I agree the documentation could do a better job of specifying the Role spec. In the meantime, I see that the Jaeger team releases a Helm chart for the Jaeger operator (GitHub repo: helm-charts) and they have defined a Role Helm template here which could be useful

[ViliusS]

For me it's also not clear. So, do I need to change all ClusterRole and ClusterRoleBindings occurrences in the manifest to Role/RoleBindings, or just specific ones? For example, do I need to change ClusterRole for jaeger-operator-metrics-reader?

Also, what about Role:manager-role definition itself? I see that it has clusterrolebindings permissions on rbac.authorization.k8s.io. Should I change that permission to rolebindings?

Helm chart only modifies manager-role and nothing else.

[sereneshikari]

Hi @ViliusS, if your security posture allows, I suggest running the Jaeger operator in cluster-wide mode itself. There are some issues I observed when running in non-cluster-wide mode that I documented in #2085

[ViliusS]

It looks like documentation here is wrong. It references jeager-operator role name however, if Jeager is installed via static yaml file the role name is manager-role.