Istio sidecar injection breaks operator

[moh-abk]

Expected Behavior

by adding the below we shouldn't see any breaking of the Jenkins operator/jenkins

---
apiVersion: v1
kind: Namespace
metadata:
  name: jenkins
    istio-injection: enabled

Jenkins and operator were running perfectly prior to the change.

Actual Behavior

Jenkins operator isn't able to start jenkins cr, logs below;

2020-05-06T00:51:17.618Z    INFO    controller-jenkins  base/reconcile.go:554   Jenkins pod volumes have changed, actual '[{jenkins-home {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {scripts {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:jenkins-operator-scripts-jenkins,},Items:[]KeyToPath{},DefaultMode:*511,Optional:nil,} nil nil nil nil nil nil nil nil nil}} {init-configuration {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:jenkins-operator-init-configuration-jenkins,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil}} {operator-credentials {nil nil nil nil nil &SecretVolumeSource{SecretName:jenkins-operator-credentials-jenkins,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {jenkins-operator-jenkins-token-wbmrc {nil nil nil nil nil &SecretVolumeSource{SecretName:jenkins-operator-jenkins-token-wbmrc,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {istio-envoy {nil &EmptyDirVolumeSource{Medium:Memory,SizeLimit:,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {podinfo {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &DownwardAPIVolumeSource{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:labels,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.labels,},ResourceFieldRef:nil,Mode:nil,},DownwardAPIVolumeFile{Path:annotations,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.annotations,},ResourceFieldRef:nil,Mode:nil,},},DefaultMode:*420,} nil nil nil nil nil nil nil nil nil nil nil nil}} {istio-token {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:istio-ca,ExpirationSeconds:*43200,Path:istio-token,},},},DefaultMode:*420,} nil nil nil nil}} {istiod-ca-cert {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:istio-ca-root-cert,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil}}]' required '[]'  {"cr": "jenkins"}
2020-05-06T00:51:17.618Z    INFO    controller-jenkins  base/reconcile.go:554   Jenkins amount of containers has changed, actual '2' required '1'   {"cr": "jenkins"}
2020-05-06T00:51:17.618Z    INFO    controller-jenkins  base/reconcile.go:554   Container '{Name:istio-proxy Image:docker.io/istio/proxyv2:1.5.2 Command:[] Args:[proxy sidecar --domain $(POD_NAMESPACE).svc.cluster.local --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster jenkins-operator.$(POD_NAMESPACE) --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istiod.istio-system.svc:15012 --zipkinAddress zipkin.istio-system:9411 --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --connectTimeout 10s --proxyAdminPort 15000 --concurrency 2 --controlPlaneAuthPolicy NONE --dnsRefreshRate 300s --statusPort 15020 --trust-domain=cluster.local --controlPlaneBootstrap=false] WorkingDir: Ports:[{Name:http-envoy-prom HostPort:0 ContainerPort:15090 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:JWT_POLICY Value:third-party-jwt ValueFrom:nil} {Name:PILOT_CERT_PROVIDER Value:istiod ValueFrom:nil} {Name:CA_ADDR Value:istio-pilot.istio-system.svc:15012 ValueFrom:nil} {Name:POD_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:POD_NAMESPACE Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:INSTANCE_IP Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:SERVICE_ACCOUNT Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.serviceAccountName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:HOST_IP Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.hostIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:ISTIO_META_POD_PORTS Value:[
    {"name":"http","containerPort":8080,"protocol":"TCP"}
    ,{"name":"slavelistener","containerPort":50000,"protocol":"TCP"}
] ValueFrom:nil} {Name:ISTIO_META_APP_CONTAINERS Value:[
    jenkins-master
] ValueFrom:nil} {Name:ISTIO_META_CLUSTER_ID Value:Kubernetes ValueFrom:nil} {Name:ISTIO_META_POD_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.name,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:ISTIO_META_CONFIG_NAMESPACE Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:ISTIO_META_INTERCEPTION_MODE Value:REDIRECT ValueFrom:nil} {Name:ISTIO_METAJSON_ANNOTATIONS Value:{"kubernetes.io/psp":"eks.privileged","sidecar.istio.io/status":"{\"version\":\"fca84600f9d5ec316cf1cf577da902f38bac258ab0fd595ee208ec0203dc0c6d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"podinfo\",\"istio-token\",\"istiod-ca-cert\"],\"imagePullSecrets\":null}"}
 ValueFrom:nil} {Name:ISTIO_META_WORKLOAD_NAME Value:jenkins-jenkins ValueFrom:nil} {Name:ISTIO_META_OWNER Value:kubernetes://apis/v1/namespaces/jenkins/pods/jenkins-jenkins ValueFrom:nil} {Name:ISTIO_META_MESH_ID Value:cluster.local ValueFrom:nil} {Name:ISTIO_KUBE_APP_PROBERS Value:{"/app-health/jenkins-master/livez":{"httpGet":{"path":"/app-health/jenkins-master/livez","port":15020,"scheme":"HTTP"},"timeoutSeconds":5},"/app-health/jenkins-master/readyz":{"httpGet":{"path":"/app-health/jenkins-master/readyz","port":15020,"scheme":"HTTP"},"timeoutSeconds":1}} ValueFrom:nil}] Resources:{Limits:map[cpu:{i:{value:2 scale:0} d:{Dec:<nil>} s:2 Format:DecimalSI} memory:{i:{value:1073741824 scale:0} d:{Dec:} s:1Gi Format:BinarySI}] Requests:map[cpu:{i:{value:100 scale:-3} d:{Dec:} s:100m Format:DecimalSI} memory:{i:{value:134217728 scale:0} d:{Dec:} s: Format:BinarySI}]} VolumeMounts:[{Name:istiod-ca-cert ReadOnly:false MountPath:/var/run/secrets/istio SubPath: MountPropagation: SubPathExpr:} {Name:istio-envoy ReadOnly:false MountPath:/etc/istio/proxy SubPath: MountPropagation: SubPathExpr:} {Name:istio-token ReadOnly:false MountPath:/var/run/secrets/tokens SubPath: MountPropagation: SubPathExpr:} {Name:podinfo ReadOnly:false MountPath:/etc/istio/pod SubPath: MountPropagation: SubPathExpr:} {Name:jenkins-operator-jenkins-token-wbmrc ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation: SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz/ready,Port:{0 15020 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:1,TimeoutSeconds:1,PeriodSeconds:2,SuccessThreshold:1,FailureThreshold:30,} StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[ALL],},Privileged:*false,SELinuxOptions:nil,RunAsUser:*1337,RunAsNonRoot:*true,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:*false,RunAsGroup:*1337,ProcMount:nil,WindowsOptions:nil,} Stdin:false StdinOnce:false TTY:false}' not found in pod {"cr": "jenkins"}

Steps to Reproduce the Problem

1. Deploy operator in jenkins-operator namespace
2. Deploy jenkins in jenkins namespace
3. Update jenkins namespace to allow istio automatic sidecar injection

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:35:15Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

• Jenkins Operator version:

v0.4.0

[tomaszsek]

195 will fix the issue.

[moh-abk]

any update on this @tomaszsek

[snooyen]

Any timeline on this? We're also looking to inject a sidecar proxy in our Jenkins pod.

[Aswartha-Rupa]

Team, any update on this? @tomaszsek

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this issue is still affecting you, just comment with any updates and we'll keep it open. Thank you for your contributions.

[dashashutosh24]

@Sig00rd Is there any progress on this? We need to add istio sidecars to make Jenkins usable for our production.

[brokenpip3]

@Sig00rd Is there any progress on this? We need to add istio sidecars to make Jenkins usable for our production.

@Sig00rd is not part of the maintainers anymore :(

I can see from the code that the deployment instead of pod is supported (even if does not seems officially released as option). You can try it via an annotation on the jenkins crd, check this code: https://github.com/jenkinsci/kubernetes-operator/blob/44a7d2460a97a74ae1dd21e286a342bec805c794/pkg/configuration/base/reconciler.go#L129

Let me know

[dashashutosh24]

@brokenpip3 Is this properly tested? I hava added the annotation to jenkins master CR, but still the operator creates a pod instead of deployment.

[brokenpip3]

Was possible in the past: https://github.com/jenkinsci/kubernetes-operator/issues/361#issuecomment-1278270760 I need to understand why is not working atm. To fix this I would like to try a different path: in the reconciliation loop skip a container in the pod if the name is istio-proxy or linkerd-proxy etc.

Which version of istio are you using and what is the name of the injected istio container?

[dashashutosh24]

@brokenpip3 Sorry for the late response. The injected container's name is istio-proxy and we are using version:1.14.6