Istio-sidecar-injector pod startup failure

[vnugent]

Description

The ansible job ran to completion but istio-sidecar-injector pod failed to start due to permission error. cc @gbaufake

master-config:

admissionConfig:
  pluginConfig:
...
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig

# oc get pods
NAME                                          READY     STATUS             RESTARTS   AGE
istio-citadel-69cc84849c-tskdb                1/1       Running            0          16m
istio-egressgateway-7f8bbcbc4f-xrgsr          1/1       Running            0          16m
istio-ingress-7d945799fc-gnknh                1/1       Running            0          16m
istio-ingressgateway-7f6d5ccc65-xvdm2         1/1       Running            0          16m
istio-pilot-578b974bcc-gphq5                  2/2       Running            0          16m
istio-policy-b5bf474cc-kmwxn                  2/2       Running            0          16m
istio-sidecar-injector-57c6b96dc4-47cc4       0/1       CrashLoopBackOff   9          16m
istio-statsd-prom-bridge-6dbb7dcc7f-kjs9p     1/1       Running            0          16m
istio-telemetry-9445d68d5-7pbk9               2/2       Running            0          16m
openshift-ansible-istio-installer-job-sm488   0/1       Completed          0          17m
prometheus-586d95b8d9-97r6v                   1/1       Running            0          16m

# oc log -f istio-sidecar-injector-57c6b96dc4-47cc4
W0622 15:42:59.841060  123350 cmd.go:358] log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-06-22T19:39:29.842384Z info    version redhat@redhat-docker.io/openshiftistio-0.8.0-6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84-Clean
2018-06-22T19:39:29.853598Z info    New configuration: sha256sum fae3fe3c7b0fbb7ee2ac5f3555d73214d1bce510a46cc8a31fbf9f9f077115b0
2018-06-22T19:39:29.853645Z info    Policy: disabled
2018-06-22T19:39:29.853704Z info    Template: |
  initContainers:
  - name: istio-init
    image: docker.io/openshiftistio/proxy-init-centos7:0.8.0
    args:
    - "-p"
    - [[ .MeshConfig.ProxyListenPort ]]
    - "-u"
    - 1337
    - "-m"
    - [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    - "-i"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeOutboundIPRanges"  ]]"
    [[ else -]]
    - "*"
    [[ end -]]
    - "-x"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeOutboundIPRanges"  ]]"
    [[ else -]]
    - ""
    [[ end -]]
    - "-b"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/includeInboundPorts"  ]]"
    [[ else -]]
    - [[ range .Spec.Containers -]][[ range .Ports -]][[ .ContainerPort -]], [[ end -]][[ end -]][[ end]]
    - "-d"
    [[ if (isset .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts") -]]
    - "[[ index .ObjectMeta.Annotations "traffic.sidecar.istio.io/excludeInboundPorts" ]]"
    [[ else -]]
    - ""
    [[ end -]]
    imagePullPolicy: IfNotPresent
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
      privileged: true
    restartPolicy: Always

  containers:
  - name: istio-proxy
    image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
    "[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
    [[ else -]]
    docker.io/openshiftistio/proxyv2-centos7:0.8.0
    [[ end -]]
    args:
    - proxy
    - sidecar
    - --configPath
    - [[ .ProxyConfig.ConfigPath ]]
    - --binaryPath
    - [[ .ProxyConfig.BinaryPath ]]
    - --serviceCluster
    [[ if ne "" (index .ObjectMeta.Labels "app") -]]
    - [[ index .ObjectMeta.Labels "app" ]]
    [[ else -]]
    - "istio-proxy"
    [[ end -]]
    - --drainDuration
    - [[ formatDuration .ProxyConfig.DrainDuration ]]
    - --parentShutdownDuration
    - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
    - --discoveryAddress
    - [[ .ProxyConfig.DiscoveryAddress ]]
    - --discoveryRefreshDelay
    - [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
    - --zipkinAddress
    - [[ .ProxyConfig.ZipkinAddress ]]
    - --connectTimeout
    - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
    - --statsdUdpAddress
    - [[ .ProxyConfig.StatsdUdpAddress ]]
    - --proxyAdminPort
    - [[ .ProxyConfig.ProxyAdminPort ]]
    - --controlPlaneAuthPolicy
    - [[ .ProxyConfig.ControlPlaneAuthPolicy ]]
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: ISTIO_META_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: ISTIO_META_INTERCEPTION_MODE
      value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
    imagePullPolicy: IfNotPresent
    securityContext:
        privileged: false
        readOnlyRootFilesystem: true
        [[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
        capabilities:
          add:
          - NET_ADMIN
        [[ else -]]
        runAsUser: 1337
        [[ end -]]
    restartPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

    volumeMounts:
    - mountPath: /etc/istio/proxy
      name: istio-envoy
    - mountPath: /etc/certs/
      name: istio-certs
      readOnly: true
  volumes:
  - emptyDir:
      medium: Memory
    name: istio-envoy
  - name: istio-certs
    secret:
      optional: true
      [[ if eq .Spec.ServiceAccountName "" -]]
      secretName: istio.default
      [[ else -]]
      secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
      [[ end -]]
2018-06-22T19:39:29.855413Z warn    Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018-06-22T19:39:29.889872Z error   Register webhook failed: mutatingwebhookconfigurations.admissionregistration.k8s.io "istio-sidecar-injector" is forbidden: User "system:serviceaccount:istio-system:istio-sidecar-injector-service-account" cannot get mutatingwebhookconfigurations.admissionregistration.k8s.io at the cluster scope: User "system:serviceaccount:istio-system:istio-sidecar-injector-service-account" cannot get mutatingwebhookconfigurations.admissionregistration.k8s.io at the cluster scope. Retrying...

Version

[knrc]

@vnugent How did you install this?

[knrc]

@vnugent Can you also send me the log from the installation job?

[vnugent]

I'm using the ansible downstream install in our test lab. Attached is ansible installer log. openshift-ansible-istio-installer-job.zip

[knrc]

This seems to be an older version of the installer and, with lots of errors during creation about resources already existing, it looks as if the issue relates to installation on an unclean system.

The latest installer can handle that use case, it will remove all traces of previous installations before it attempts to install Istio. Can you try updating to the latest installer and giving it another try?

[vnugent]

Ansible install job v 0.8.0 worked!