Open ediezh opened 3 years ago
@ediezh could you please provide the deployment that is being used by the operator?
kubectl get deployment -n observability jaeger-operator -o yaml
I'm especially interested in the value of the WATCH_NAMESPACE
.
@jpkrohling here is the deployment.
apiVersion: v1
kind: Pod
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2021-04-16T08:10:16Z"
kubernetes.io/psp: eks.privileged
prometheus.io/path: /stats/prometheus
prometheus.io/port: "15020"
prometheus.io/scrape: "true"
sidecar.istio.io/status: '{"version":"8e6e902b765af607513b28d284940ee1421e9a0d07698741693b2663c7161c11","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-ca-cert"],"imagePullSecrets":null}'
creationTimestamp: "2021-04-16T08:10:16Z"
generateName: jaeger-operator-799c7fbbb5-
labels:
app.kubernetes.io/name: jaeger-operator
istio.io/rev: 1-7-6
pod-template-hash: 799c7fbbb5
security.istio.io/tlsMode: istio
service.istio.io/canonical-name: jaeger-operator
service.istio.io/canonical-revision: latest
name: jaeger-operator-799c7fbbb5-c7l6g
namespace: observability
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: jaeger-operator-799c7fbbb5
uid: 8b7d7de3-7bd7-4349-b4df-fb4c0d9a4c04
resourceVersion: "22781385"
selfLink: /api/v1/namespaces/observability/pods/jaeger-operator-799c7fbbb5-c7l6g
uid: db8dc1f6-a2c8-46be-ac42-d19fcd400b2d
spec:
containers:
- args:
- proxy
- sidecar
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --serviceCluster
- jaeger-operator.observability
- --proxyLogLevel=info
- --proxyComponentLogLevel=misc:error
- --trust-domain=cluster.local
- --concurrency
- "2"
env:
- name: JWT_POLICY
value: third-party-jwt
- name: PILOT_CERT_PROVIDER
value: istiod
- name: CA_ADDR
value: istiod-1-7-6.istio-system.svc:15012
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.serviceAccountName
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: CANONICAL_SERVICE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['service.istio.io/canonical-name']
- name: CANONICAL_REVISION
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['service.istio.io/canonical-revision']
- name: PROXY_CONFIG
value: |
{"discoveryAddress":"istiod-1-7-6.istio-system.svc:15012","tracing":{"zipkin":{"address":"jaeger-operator-jaeger-collector.observability.svc.cluster.local:9411"}},"proxyMetadata":{"DNS_AGENT":""}}
- name: ISTIO_META_POD_PORTS
value: |-
[
{"name":"metrics","containerPort":8383,"protocol":"TCP"}
]
- name: ISTIO_META_APP_CONTAINERS
value: jaeger-operator
- name: ISTIO_META_CLUSTER_ID
value: Kubernetes
- name: ISTIO_META_INTERCEPTION_MODE
value: REDIRECT
- name: ISTIO_METAJSON_ANNOTATIONS
value: |
{"kubectl.kubernetes.io/restartedAt":"2021-04-16T08:10:16Z","kubernetes.io/psp":"eks.privileged"}
- name: ISTIO_META_WORKLOAD_NAME
value: jaeger-operator
- name: ISTIO_META_OWNER
value: kubernetes://apis/apps/v1/namespaces/observability/deployments/jaeger-operator
- name: ISTIO_META_MESH_ID
value: cluster.local
- name: DNS_AGENT
- name: ISTIO_KUBE_APP_PROBERS
value: '{}'
image: 018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/istio/proxyv2:1.7.6
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- pilot-agent
- wait
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
readinessProbe:
failureThreshold: 30
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 100m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
privileged: true
readOnlyRootFilesystem: true
runAsGroup: 1337
runAsNonRoot: true
runAsUser: 1337
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/istio
name: istiod-ca-cert
- mountPath: /var/lib/istio/data
name: istio-data
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /var/run/secrets/tokens
name: istio-token
- mountPath: /etc/istio/pod
name: istio-podinfo
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: jaeger-operator-token-4n56t
readOnly: true
- args:
- start
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: OPERATOR_NAME
value: jaeger-operator
image: 018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/jaegertracing/jaeger-operator:1.22.0
imagePullPolicy: IfNotPresent
name: jaeger-operator
ports:
- containerPort: 8383
name: metrics
protocol: TCP
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: jaeger-operator-token-4n56t
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- args:
- istio-iptables
- -p
- "15001"
- -z
- "15006"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- '*'
- -d
- 15090,15021,15020
env:
- name: DNS_AGENT
image: 018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/istio/proxyv2:1.7.6
imagePullPolicy: IfNotPresent
name: istio-init
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 10m
memory: 10Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_ADMIN
- NET_RAW
drop:
- ALL
privileged: true
readOnlyRootFilesystem: false
runAsGroup: 0
runAsNonRoot: false
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: jaeger-operator-token-4n56t
readOnly: true
nodeName: ip-10-202-118-5.ap-northeast-2.compute.internal
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1337
serviceAccount: jaeger-operator
serviceAccountName: jaeger-operator
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: jaeger-operator-token-4n56t
secret:
defaultMode: 420
secretName: jaeger-operator-token-4n56t
- emptyDir:
medium: Memory
name: istio-envoy
- emptyDir: {}
name: istio-data
- downwardAPI:
defaultMode: 420
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
path: annotations
name: istio-podinfo
- name: istio-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: istio-ca
expirationSeconds: 43200
path: istio-token
- configMap:
defaultMode: 420
name: istio-ca-root-cert
name: istiod-ca-cert
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-04-16T08:10:18Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-04-16T08:10:21Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-04-16T08:10:21Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-04-16T08:10:16Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://4aa62e1e71bd4ba4404a684e806eaa36ab91ed94fa10315fd6b3cb1ffc6d5052
image: 018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/istio/proxyv2:1.7.6
imageID: docker-pullable://018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/istio/proxyv2@sha256:e526549f64ebcc3436de9f92ae322856dd02bd59f58c3bc034d31f9fce45d6a0
lastState: {}
name: istio-proxy
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2021-04-16T08:10:18Z"
- containerID: docker://c18a1846759cd41f03b672db0dcb414f92e2b0d4dbd13d19f23ad0a4de20ca15
image: 018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/jaegertracing/jaeger-operator:1.22.0
imageID: docker-pullable://018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/jaegertracing/jaeger-operator@sha256:da7f9035f3a35f86d5f936e2720608d84ceae127ddbfcb8f78694c80fa678927
lastState: {}
name: jaeger-operator
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2021-04-16T08:10:20Z"
hostIP: 10.202.118.5
initContainerStatuses:
- containerID: docker://c67d71f910e720baa9f5654a336de0d7fbdecd66f47271a8015b134218a3ca4f
image: 018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/istio/proxyv2:1.7.6
imageID: docker-pullable://018661664346.dkr.ecr.ap-northeast-2.amazonaws.com/istio/proxyv2@sha256:e526549f64ebcc3436de9f92ae322856dd02bd59f58c3bc034d31f9fce45d6a0
lastState: {}
name: istio-init
ready: true
restartCount: 0
state:
terminated:
containerID: docker://c67d71f910e720baa9f5654a336de0d7fbdecd66f47271a8015b134218a3ca4f
exitCode: 0
finishedAt: "2021-04-16T08:10:17Z"
reason: Completed
startedAt: "2021-04-16T08:10:17Z"
phase: Running
podIP: 10.202.100.133
podIPs:
- ip: 10.202.100.133
qosClass: Burstable
startTime: "2021-04-16T08:10:16Z"
It does look like a bug to me: the operator shouldn't be looking at all namespaces when it's restricted to one.
@rkukura, are you able to take a look at this one?
ping @rkukura, are you able to work on this soon?
FWIW, in addition to the error about listing namespaces, I also got secrets is forbidden: User \"system:serviceaccount:default:my-release-jaeger-operator\" cannot list resource \"secrets\" in API group \"\" at the cluster scope
with 1.22.1
when deployed via the official Helm chart.
bumping this as the current operator with create enabled does not install the CR. Setting rbac.clusterRole: true
does not help as there is no mention of namespace
in the RBAC delivered with the chart.
Adding this little snippet to the role will fix it:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- list
- watch
@Morriz will you open PR about it?
sure
I'm not sure but get
verbs is needed for namespaces
- apiGroups:
- ""
resources:
- namespaces
verbs:
- list
- watch
- get
don't think so. I saw no more errors without it...
We have this already listed in the cluster role:
Is that not sufficient?
I have a feeling some of the conversation here might be confusing things a bit, since there was a period of time where both modes had some problems on various versions of the Helm chart and operator. The original ticket was about when not running in cluster mode, i.e when WATCH_NAMESPACE
is defined and points to a single namespace.
It seemed the operator itself still tried to list namespaces which yielded this error, and which in theory it should not need to do or attempt to do when it was running in single namespace mode, since to do so would require the ClusterRole
which you wouldn't normally be deploying when in this mode.
Indeed the official Helm chart will not create ClusterRole
s and ClusterRoleBinding
s when in this mode: https://github.com/jaegertracing/helm-charts/blob/59f51fd7caf924faaea951cf064f86c3f35e8b78/charts/jaeger-operator/templates/role.yaml#L2
... using the same value in the chart which controls "single namespace mode" on the Deployment
.
https://github.com/jaegertracing/helm-charts/blob/59f51fd7caf924faaea951cf064f86c3f35e8b78/charts/jaeger-operator/templates/deployment.yaml#L43-L50
Nevertheless, that PR doesn't seem like it'd fix this issue to me since it is touching the ClusterRole
.
I looked into this to see if I could fix it but it is above my skill level with sigs.k8s.io/controller-runtime.
_jaeger-operator/pkg/controller/namespace/namespacecontroller.go:52 sets up a Watch()
for namespaces so it can wake up when sidecar.jaegertracing.io/inject
changes.
It would be straightforward to skip watching namespaces if $WATCH_NAMESPACE
is exactly one namespace but then we lose the ability to respond to changes in the annotation, even in the watched namespace.
I couldn't figure out how use Controller.Watch() for anything less than all namespaces. I didn't chase this problem for long because I suspect there is no solution -- I suspect controller-runtime Watch()
requires RBAC to list. (I would love to be wrong!). The possibility to watch a single namespace using the go-client exists (see https://github.com/kubernetes/kubernetes/issues/43299 ) but using a totally different technique to watch for annotations seems too complex for this problem.
While working on this I noticed that if the Jaeger Operator has permission to list the namespaces it will reconcile ALL the namespaces, not just $WATCH_NAMESPACE. Set the operator to log at trace level, run locally with WATCH_NAMESPACE=observability make run-debug
and you'll see it claim to reconcile all of the namespaces. Can someone confirm if this is unintended behavior and file an issue to track? WATCH_NAMESPACE should constrain both looking for Jaeger CRs and looking for Jaeger annotations, no?
I suspect there is an additional problem that needs to be fixed. (This may need its own issue or perhaps it can be part of this issue.)
If Jaeger Operator service account has no clusterrolebinding to LIST namespaces, I suspect it will fail to utilize sidecar.jaegertracing.io/inject
namespace annotations, even for the namespace it is running within.
@ediezh Work around for the message. The warning should be suppressed if the operator is started with --enable-namespace-controller=false
.
I am facing the same issue .
User "system:serviceaccount:jaeger-operator:jaeger-operator" cannot list resource "namespaces" in API group "" at the cluster scope
hi @s9r-5, what jaeger operator version do you use?
Guys, the same issue.
Deployed jaeger-operator 2.39.0 from helm chart using default values (non-cluster wide) in observability
ns. When I try to create the simplest jaeger cluster I get:
1.6744841582063437e+09 ERROR Reconciler error {"controller": "jaeger", "controllerGroup": "jaegertracing.io", "controllerKind": "Jaeger", "Jaeger": {"name":"simplest","namespace":"observability"}, "namespace": "observability", "name": "simplest", "reconcileID": "44701060-9d41-42ad-b849-dfe6f916d60b", "error": "deployments.apps is forbidden: User \"system:serviceaccount:observability:jaeger-crd-jaeger-operator\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope"}
For some reason operator wants to list deployments in cluster scope but I don't use cluster scope. I can't deploy even simplest jaeger cluster.
I am seeing these errors in my logs, but also maybe the deployments are getting provisioned?
I am using jaeger-operator installed from the helm-chart, version 2.49.0. I am using all the defaults in values.yaml and installed into the namespace 'observability' as expected.
Is this repo still active? This issue has been active since April 2021 and it seems it is not getting enough traction. I think it is important for the community to have this issue solved.
Jaeger should not be using a cluster-wide role to access namespaces if we explicitly don't want to use this role, and this means also that Jaeger is not following the principle of least privilege. Why would Jaeger require more permissions than it needs?
Is this repo still active? This issue has been active since April 2021 and it seems it is not getting enough traction. I think it is important for the community to have this issue solved.
It is. We will be pleased to review a PR with a solution.
Deployed the operator 1.22 following these instructions
The operator watches only the observability namespace. But I'm seeing these errors in the log: