search

jaegertracing / helm-charts

Helm Charts for Jaeger backend
Apache License 2.0
269 stars 347 forks source link

[Bug]: Jaeger Custom Resource not works #581

Closed Dante-in-Korea closed 3 months ago

Dante-in-Korea commented 5 months ago

What happened?

I've installed jaeger-oprator latest version. and then when I deploy jaeger resource, jaeger-collector and jaeger-query deployment resources don't create in kubernetes.

I've reinstalled jaeger-operator v2.53.0 and jaeger resource. It's been working very well.

but jaeger-operator v2.54.0 is not working.. Below is jaeger-operator's logs.

W0530 01:01:26.187031       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:jaeger:jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

E0530 01:01:26.187071       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User 
"system:serviceaccount:jaeger:jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

It seems that Jaeger service accounts don't have the ingressClasses rules in the clusterrole. I've added it to the clusterrole resources, and as a result, jaeger-collector and jaeger-query deployments have been created!

Steps to reproduce

Install the Jaeger-operator v2.54.0

cat <<EOF > values.yaml 
# Default values for jaeger-operator.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

image:
  repository: jaegertracing/jaeger-operator
  tag: 1.52.0
  pullPolicy: IfNotPresent
  imagePullSecrets: []

certs:
  issuer:
    create: true
    name: ""
  certificate:
    create: true
    namespace: ""
    secretName: ""
    issuerKind: Issuer

webhooks:
  mutatingWebhook:
    create: true
  validatingWebhook:
    create: true
  port: 9443
  service:
    annotations: {}
    create: true
    name: ""

jaeger:
  create: false
  namespace:
  spec: {}

rbac:
  create: true
  pspEnabled: false
  clusterRole: true

service:
  type: ClusterIP
  annotations: {}

serviceAccount:
  create: true
  name:
  annotations: {}

extraArgs: {}

extraEnv: []

serviceExtraLabels: {}

extraLabels: {}

replicaCount: 1

resources: {}

nodeSelector: {}

tolerations: []

topologySpreadConstraints: []

affinity: {}

securityContext: {}

containerSecurityContext: {}

priorityClassName:

hostNetwork: true

metricsPort: 8383

EOF 
helm search repo jaegertracing/jaeger-operator
helm upgrade --install -n jaeger jaeger-operator jaegertracing/jaeger-operator --version 2.54.0 -f values.yaml

Create the jaeger custom resource file. 

cat <<EOF > jaeger-es.yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-prod
  namespace: jaeger
  labels:
    jaegertracing.io/operated-by: jaeger.jaeger-operator
    manager: jaeger-operator
spec: 
  strategy: production
  ingress:
    enabled: false
  storage:
    type: elasticsearch 
    options:
      es:
        server-urls: https://opensearch-cluster-master.opensearch.svc.cluster.local:9200 
        index-prefix: jaeger
        tls.skip-host-verify: true
    secretName: jaeger-secret 
EOF

Create the Jaeger custom resource

kubectl apply -f  jaeger-es.yaml

Check the Jaeger resource there's no the jaeger-collector, jaeger-query

kubectl -n jaeger get jaeger
NAME          STATUS   VERSION   STRATEGY     STORAGE         AGE
jaeger-prod                      production   elasticsearch   49s
kubectl -n jaeger get all
NAME                                   READY   STATUS    RESTARTS   AGE
pod/jaeger-operator-668cf4cf99-mlbdf   1/1     Running   0          100s

NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/jaeger-operator-metrics           ClusterIP   10.99.114.177   <none>        8383/TCP   47m
service/jaeger-operator-webhook-service   ClusterIP   10.108.24.208   <none>        443/TCP    47m

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger-operator   1/1     1            1           47m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/jaeger-operator-668cf4cf99   1         1         1       47m

Check the Jaeger-operator logs

W0530 01:47:32.688557       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:jaeger:jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0530 01:47:32.688592       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:jaeger:jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

Add the ingressclasses rules in jaeger-operator clusterrole

kubectl edit clusterrole jaeger-operator

Add the content below.

- apiGroups: 
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

Check the Jaeger resources A few minute later, you can see the jaeger-collector and Jaeger-query resources.

kubectl -n jaeger get pod
NAME                                     READY   STATUS    RESTARTS   AGE
jaeger-operator-668cf4cf99-mlbdf         1/1     Running   0          14m
jaeger-prod-collector-7d69b54666-mmrcd   1/1     Running   0          65s
jaeger-prod-query-9bd6465df-kljjz        2/2     Running   0          65s

It works excellently as anticipated.

Expected behavior

When I set rbac.cluster.Role to true, it should work find without editing the jaeger-operator cluster role.

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

v1.57.0

SDK

No response

Pipeline

No response

Stogage backend

opensearch

Operating system

Linux

Deployment model

Kubernetes

Deployment configs

No response

martygeo commented 5 months ago

I'm having the same problem also using chart version 2.54.0 (upgrading to jaeger-operator v1.57)

kravciak commented 5 months ago

I can confirm this, maybe related to https://github.com/jaegertracing/jaeger-operator/commit/c8b3e7351482236f63f18da52a8292a27fdf890e?

Dante-in-Korea commented 4 months ago

@kravciak sorry, I checked it late. I guess there is no the clusterrole resource in there.

now I've re-checked. you've set the route.openshift.io/routes/custom-host resource in "bundle/manifests/jaeger-operator.clusterserviceversion.yaml". I can't find this role when I install jaeger-operator.

kubectl get clusterrole jaeger-operator -o jsonpath='{.rules[?(@.apiGroups[0] == "route.openshift.io")]}'
{"apiGroups":["route.openshift.io"],"resources":["routes"],"verbs":["create","delete","get","list","patch","update","watch"]}%

there is no the routes/custom-host resource in jaeger-operator clusterrole.

Dante-in-Korea commented 4 months ago

I think you should change charts/jaeger-operator/templates/role.yaml

it doesn't have the ingressclasses and routes/custom-host resource. https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger-operator/templates/role.yaml