search

jenkinsci / helm-charts

Jenkins helm charts
https://artifacthub.io/packages/helm/jenkinsci/jenkins
Apache License 2.0
562 stars 890 forks source link

Helm repo jenkins/jenkins "CrashLoopBackOff" with config-reload container error "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f4ec8b10250>: Failed to establish a new connection: [Errno 111] Connection refused'))" #222

Closed hasakura12 closed 3 years ago

hasakura12 commented 3 years ago

Describe the bug helm install jenkins jenkins/jenkins -n jenkins -f overrides.yaml fails with CrashLoopBackOff. Running on AWS EKS 1.8 and istio 1.7.

I checked the similar issue https://github.com/jenkinsci/helm-charts/issues/185 but my case is a bit different as init-container succeeds without errors.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}```

Kubernetes Version:

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:08:32Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
$ istioctl version
1.7.0
data plane version: 1.7.0 (45 proxies)

Which version of the chart: jenkins/jenkins:2.263.2

What happened:

helm repo update
helm repo add jenkins https://charts.jenkins.io
helm install jenkins jenkins/jenkins     -n jenkins     -f overrides.yaml

overrides.yaml contents

controller:
  # use Docker in Docker jenkins, so that jenkins container can build docker image inside
  # image: mesosphere/jenkins-dind # https://hub.docker.com/r/mesosphere/jenkins-dind
  # tag: 0.9.0
  statefulSetLabels:
    app: jenkins  # needed for istio
    version: 2.0.0  # needed for istio
  serviceLabels:
    app: jenkins  # needed for istio
    version: 2.0.0  # needed for istio
  podLabels:
    app: jenkins  # needed for istio
    version: 2.0.0  # needed for istio
  additionalPlugins: # WARNING: uncommenting out these will cause pod to crash due to "cp -r not specified". So for now, these plugins need be installed manually
    # - kubernetes:1.25.7
    # - workflow-job:2.39
    # - workflow-aggregator:2.6
    # - credentials-binding:1.23
    # - git:4.2.2
    # - configuration-as-code:1.41
    # - bitbucket:.1.1.11 # https://plugins.jenkins.io/bitbucket/
    # - bitbucket-build-status-notifier:1.4.2 # https://plugins.jenkins.io/bitbucket-build-status-notifier/
    # - bitbucket-oauth:0.10
    # - docker-build-publish:1.554.2  # https://plugins.jenkins.io/docker-build-publish/
    # - amazon-ecr:1.6 # https://plugins.jenkins.io/amazon-ecr/
    # - slack:2.40 # https://plugins.jenkins.io/slack/
    # - blueocean:1.23.2 # https://plugins.jenkins.io/blueocean/
    # - disk-usage:0.28 # https://plugins.jenkins.io/disk-usage/
    # - ws-cleanup:0.38 # https://plugins.jenkins.io/ws-cleanup/
    # - timestamper:1.11.3 # https://plugins.jenkins.io/timestamper/
    # - build-timeout:1.20 # https://plugins.jenkins.io/build-timeout/
  JCasC: # refs: https://github.com/helm/charts/tree/master/stable/jenkins#configuration-as-code-new-default, https://github.com/helm/charts/issues/22693
    enabled: false # disable this because of errors getting (see logs/jenkings.log, log ID=ID 68acadb6-48d2-4341-97bf-bd3e4300a71e)
    defaultConfig: false

serviceAccount:
  name: jenkins
  # for Jenkins pod to assume IAM role (IRSA)
  annotations: 
    eks.amazonaws.com/role-arn: "arn:aws:iam::xxxxxxxxx:role/EKSJenkinsRole"

persistence:
  existingClaim: jenkins-claim # efs csi driver doesn't support dynamic provisioning, so pv and pvc needs to be precreated. Ref: https://github.com/kubernetes-sigs/aws-efs-csi-driver
  # storageClass: efs # use EFS storageclass. If the storage class is set to null or left undefined (persistence.storageClass=), the default provisioner is used (gp2 on AWS, standard on GKE, AWS & OpenStack).
  size: 8Gi

$ k get pod -n jenkins
NAME        READY   STATUS             RESTARTS   AGE
jenkins-0   1/3     CrashLoopBackOff   12         7m18s
$ k describe pod jenkins-0 -n jenkins
Name:         jenkins-0
Namespace:    jenkins
Priority:     0
Node:         ip-10-1-104-142.ec2.internal/10.1.104.142
Start Time:   Tue, 19 Jan 2021 03:54:50 +0700
Labels:       app=jenkins
              app.kubernetes.io/component=jenkins-controller
              app.kubernetes.io/instance=jenkins
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=jenkins
              controller-revision-hash=jenkins-7dfdd99d7
              istio.io/rev=default
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=jenkins
              service.istio.io/canonical-revision=2.0.0
              statefulset.kubernetes.io/pod-name=jenkins-0
              version=2.0.0
Annotations:  checksum/config: 2a4c2b3ea5dea271cb7c0b8e8582b682814d39f8e933e0348725b0b9a7dbf258
              kubernetes.io/psp: eks.privileged
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/status:
                {"version":"8e6e902b765af607513b28d284940ee1421e9a0d07698741693b2663c7161c11","initContainers":["istio-init"],"containers":["istio-proxy"]...
Status:       Running
IP:           10.1.104.213
IPs:
  IP:           10.1.104.213
Controlled By:  StatefulSet/jenkins
Init Containers:
  init:
    Container ID:  docker://a8a905190d4a5e918b3f98ad4161abba2164a41454c25985b097178da578046e
    Image:         jenkins/jenkins:2.263.2
    Image ID:      docker-pullable://jenkins/jenkins@sha256:63fde6791bc9e11dc8fc94830ac76228afbe8d31206ab8f83f251314a3f45ba9
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 19 Jan 2021 03:54:53 +0700
      Finished:     Tue, 19 Jan 2021 03:55:04 +0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:     50m
      memory:  256Mi
    Environment:
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::xxxxxxx:role/EKSJenkinsRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /usr/share/jenkins/ref/plugins from plugins (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_plugins from plugin-dir (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-hflcp (ro)
  istio-init:
    Container ID:  docker://b39094acf40625ff7fbbf7416352f9a91776df8fb83cda748756d805fb8bc332
    Image:         docker.io/istio/proxyv2:1.7.0
    Image ID:      docker-pullable://istio/proxyv2@sha256:c1f1b45a4162509f86aa82d0148aef55824454e7204f27f23dddc9d7f4ae7cd1
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 19 Jan 2021 03:55:06 +0700
      Finished:     Tue, 19 Jan 2021 03:55:06 +0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     10m
      memory:  10Mi
    Environment:
      DNS_AGENT:                    
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::xxxxxxx:role/EKSJenkinsRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-hflcp (ro)
Containers:
  jenkins:
    Container ID:  docker://a3eebbd8e056d2223e102eb9bd99b72ba08ee1ccfcb87b803483acf86d934192
    Image:         jenkins/jenkins:2.263.2
    Image ID:      docker-pullable://jenkins/jenkins@sha256:63fde6791bc9e11dc8fc94830ac76228afbe8d31206ab8f83f251314a3f45ba9
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --httpPort=8080
    State:          Running
      Started:      Tue, 19 Jan 2021 03:55:57 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Tue, 19 Jan 2021 03:55:07 +0700
      Finished:     Tue, 19 Jan 2021 03:55:56 +0700
    Ready:          False
    Restart Count:  1
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      50m
      memory:   256Mi
    Liveness:   http-get http://:15020/app-health/jenkins/livez delay=0s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:15020/app-health/jenkins/readyz delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:                     jenkins-0 (v1:metadata.name)
      JAVA_OPTS:                    -Dcasc.reload.token=$(POD_NAME) 
      JENKINS_OPTS:                 
      JENKINS_SLAVE_AGENT_PORT:     50000
      CASC_JENKINS_CONFIG:          /var/jenkins_home/casc_configs
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::xxxxxxx:role/EKSJenkinsRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /run/secrets/chart-admin-password from admin-secret (ro,path="jenkins-admin-password")
      /run/secrets/chart-admin-username from admin-secret (ro,path="jenkins-admin-user")
      /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from sc-config-volume (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-hflcp (ro)
  config-reload:
    Container ID:   docker://970ef80b78243923d73d113b28825730e5171bf51a634673d9959854deb3d6c5
    Image:          kiwigrid/k8s-sidecar:0.1.275
    Image ID:       docker-pullable://kiwigrid/k8s-sidecar@sha256:4ef7b5f142717a1af3a4c3df4991cdca271f29a45f1d10ca68d04072a34ca814
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 19 Jan 2021 03:55:07 +0700
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:                     jenkins-0 (v1:metadata.name)
      LABEL:                        jenkins-jenkins-config
      FOLDER:                       /var/jenkins_home/casc_configs
      NAMESPACE:                    jenkins
      REQ_URL:                      http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)
      REQ_METHOD:                   POST
      REQ_RETRY_CONNECT:            10
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::xxxxxxx:role/EKSJenkinsRole
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_home/casc_configs from sc-config-volume (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-hflcp (ro)
  istio-proxy:
    Container ID:  docker://1ae2dfc3528c16d47de93e82daaa212d0ad366847a96da5daaf3e6402cf7d2ed
    Image:         docker.io/istio/proxyv2:1.7.0
    Image ID:      docker-pullable://istio/proxyv2@sha256:c1f1b45a4162509f86aa82d0148aef55824454e7204f27f23dddc9d7f4ae7cd1
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      jenkins.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --trust-domain=cluster.local
      --concurrency
      2
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Tue, 19 Jan 2021 03:55:48 +0700
      Finished:     Tue, 19 Jan 2021 03:55:48 +0700
    Ready:          False
    Restart Count:  3
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      10m
      memory:   40Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      jenkins-0 (v1:metadata.name)
      POD_NAMESPACE:                 jenkins (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])
      PROXY_CONFIG:                  {"tracing":{"datadog":{"address":"$(HOST_IP):8126"}},"proxyMetadata":{"DNS_AGENT":""}}

      ISTIO_META_POD_PORTS:          [
                                         {"name":"http","containerPort":8080,"protocol":"TCP"}
                                         ,{"name":"agent-listener","containerPort":50000,"protocol":"TCP"}
                                     ]
      ISTIO_META_APP_CONTAINERS:     jenkins,config-reload
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_METAJSON_ANNOTATIONS:    {"checksum/config":"2a4c2b3ea5dea271cb7c0b8e8582b682814d39f8e933e0348725b0b9a7dbf258","kubernetes.io/psp":"eks.privileged"}

      ISTIO_META_WORKLOAD_NAME:      jenkins
      ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/jenkins/statefulsets/jenkins
      ISTIO_META_MESH_ID:            cluster.local
      DNS_AGENT:                     
      ISTIO_KUBE_APP_PROBERS:        {"/app-health/jenkins/livez":{"httpGet":{"path":"/login","port":8080,"scheme":"HTTP"},"timeoutSeconds":5},"/app-health/jenkins/readyz":{"httpGet":{"path":"/login","port":8080,"scheme":"HTTP"},"timeoutSeconds":5},"/app-health/jenkins/startupz":{"httpGet":{"path":"/login","port":8080,"scheme":"HTTP"},"timeoutSeconds":5}}
      AWS_DEFAULT_REGION:            us-east-1
      AWS_REGION:                    us-east-1
      AWS_ROLE_ARN:                  arn:aws:iam::xxxxxx:role/EKSJenkinsRole
      AWS_WEB_IDENTITY_TOKEN_FILE:   /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-hflcp (ro)
      /var/run/secrets/tokens from istio-token (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  plugins:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jenkins
    Optional:  false
  plugin-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins-claim
    ReadOnly:   false
  sc-config-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  admin-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  jenkins
    Optional:    false
  jenkins-token-hflcp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  jenkins-token-hflcp
    Optional:    false
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        istio-ca-root-cert
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    84s                default-scheduler  Successfully assigned jenkins/jenkins-0 to ip-10-1-104-142.ec2.internal
  Warning  FailedMount  83s                kubelet            MountVolume.SetUp failed for volume "jenkins-config" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  83s                kubelet            MountVolume.SetUp failed for volume "istiod-ca-cert" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  83s                kubelet            MountVolume.SetUp failed for volume "admin-secret" : failed to sync secret cache: timed out waiting for the condition
  Normal   Pulling      82s                kubelet            Pulling image "jenkins/jenkins:2.263.2"
  Normal   Pulled       82s                kubelet            Successfully pulled image "jenkins/jenkins:2.263.2"
  Normal   Created      82s                kubelet            Created container init
  Normal   Started      82s                kubelet            Started container init
  Normal   Pulling      70s                kubelet            Pulling image "docker.io/istio/proxyv2:1.7.0"
  Normal   Pulled       70s                kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.7.0"
  Normal   Created      70s                kubelet            Created container istio-init
  Normal   Pulled       69s                kubelet            Successfully pulled image "jenkins/jenkins:2.263.2"
  Normal   Pulling      69s                kubelet            Pulling image "jenkins/jenkins:2.263.2"
  Normal   Started      69s                kubelet            Started container istio-init
  Normal   Created      68s                kubelet            Created container jenkins
  Normal   Started      68s                kubelet            Started container jenkins
  Normal   Pulled       68s                kubelet            Container image "kiwigrid/k8s-sidecar:0.1.275" already present on machine
  Normal   Created      68s                kubelet            Created container config-reload
  Normal   Started      68s                kubelet            Started container config-reload
  Normal   Started      68s                kubelet            Started container istio-proxy
  Normal   Pulling      67s (x2 over 68s)  kubelet            Pulling image "docker.io/istio/proxyv2:1.7.0"
  Normal   Pulled       66s (x2 over 68s)  kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.7.0"
  Normal   Created      66s (x2 over 68s)  kubelet            Created container istio-proxy
$ k logs jenkins-0 -n jenkins -c istio-proxy -f
2021-01-18T20:58:06.015338Z     info    FLAG: --concurrency="2"
2021-01-18T20:58:06.015377Z     info    FLAG: --disableInternalTelemetry="false"
2021-01-18T20:58:06.015387Z     info    FLAG: --domain="jenkins.svc.cluster.local"
2021-01-18T20:58:06.015393Z     info    FLAG: --help="false"
2021-01-18T20:58:06.015399Z     info    FLAG: --id=""
2021-01-18T20:58:06.015410Z     info    FLAG: --ip=""
2021-01-18T20:58:06.015419Z     info    FLAG: --log_as_json="false"
2021-01-18T20:58:06.015429Z     info    FLAG: --log_caller=""
2021-01-18T20:58:06.015438Z     info    FLAG: --log_output_level="default:info"
2021-01-18T20:58:06.015448Z     info    FLAG: --log_rotate=""
2021-01-18T20:58:06.015455Z     info    FLAG: --log_rotate_max_age="30"
2021-01-18T20:58:06.015461Z     info    FLAG: --log_rotate_max_backups="1000"
2021-01-18T20:58:06.015467Z     info    FLAG: --log_rotate_max_size="104857600"
2021-01-18T20:58:06.015472Z     info    FLAG: --log_stacktrace_level="default:none"
2021-01-18T20:58:06.015482Z     info    FLAG: --log_target="[stdout]"
2021-01-18T20:58:06.015488Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2021-01-18T20:58:06.015493Z     info    FLAG: --mixerIdentity=""
2021-01-18T20:58:06.015499Z     info    FLAG: --outlierLogPath=""
2021-01-18T20:58:06.015504Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2021-01-18T20:58:06.015518Z     info    FLAG: --proxyLogLevel="warning"
2021-01-18T20:58:06.015524Z     info    FLAG: --serviceCluster="jenkins.jenkins"
2021-01-18T20:58:06.015530Z     info    FLAG: --serviceregistry="Kubernetes"
2021-01-18T20:58:06.015535Z     info    FLAG: --stsPort="0"
2021-01-18T20:58:06.015540Z     info    FLAG: --templateFile=""
2021-01-18T20:58:06.015546Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-01-18T20:58:06.015552Z     info    FLAG: --trust-domain="cluster.local"
2021-01-18T20:58:06.015585Z     info    Version 1.7.0-2022348138e47498c4b54995b4cb5a1656817c4e-Clean
2021-01-18T20:58:06.015795Z     info    Obtained private IP [10.1.104.213]
2021-01-18T20:58:06.015905Z     info    Apply proxy config from env {"tracing":{"datadog":{"address":"10.1.104.142:8126"}},"proxyMetadata":{"DNS_AGENT":""}}

2021-01-18T20:58:06.017143Z     info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: jenkins.jenkins
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  datadog:
    address: 10.1.104.142:8126

2021-01-18T20:58:06.017193Z     info    Proxy role: &model.Proxy{Type:"sidecar", IPAddresses:[]string{"10.1.104.213"}, ID:"jenkins-0.jenkins", Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:"jenkins.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil), ActiveExperimental:map[string]*model.WatchedResource(nil), RequestedTypes:struct { CDS string; EDS string; RDS string; LDS string }{CDS:"", EDS:"", RDS:"", LDS:""}}
2021-01-18T20:58:06.017206Z     info    JWT policy is third-party-jwt
2021-01-18T20:58:06.017260Z     info    PilotSAN []string{"istiod.istio-system.svc"}
2021-01-18T20:58:06.017282Z     info    MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"}
2021-01-18T20:58:06.017345Z     info    sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2021-01-18T20:58:06.017358Z     info    Using user-configured CA istiod.istio-system.svc:15012
2021-01-18T20:58:06.017366Z     info    istiod uses self-issued certificate
2021-01-18T20:58:06.017436Z     info    the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRAK0TrRDsvbJrIuo+tJ0mAM0wDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMTAxMDYxODEwMzNaFw0zMTAx
MDQxODEwMzNaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQDm0JSJflp4uR99y8dK3KS32UyjsrQRjT/kEUpD
xaM0zMjK3Z91E7qxG6QvW36NdPN2qCNGmdL7YsVSU9gH0JjtiiYdzQUMLP6LeSHN
r64AptfKUNpPdngWEeYIf0OXOd9VjbGrb1CxWZ8+mAHQTlfDzuuySA7cSjUDOuy5
yNyQyk734RazUrgdTUyHjTT8X5OXNObCzGT8qjgKuRoCAr51a58tb6QZ0TNsz7aM
HaT/oEiZReBUKWu91tDv5hWrRlJPuGwNb/6jP0Ema2k6TURDmg9U+38TswYU9pd2
ssH2itzxxcWCdpAjUjyGlALxfppmd1Oelkt7SxEF4fK2Q71BAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQAToGFCwPgPXl6zTPsoS6tcwI7t8QDVfwLEBHWUrEaIOVssbpD6fIKf3/Bj0sKB
/70NMpaf1AhuUde5IawY7me4mC55VYe54SVmF8yVIH+iq/yKuXxCAK7Rkm8TvoFK
ce69DhgsIETMuSTHFKWg77CZhMxdZsU9HBrORJAui2vl5tOlXpHwdnIDOD2zEf+F
dLiAWhvZiMRc5BLqsPjwoqTLLeFuV/CM2eQU4wdPqRFsx3F17Z32mBMklkAZpHh2
24notqa6Zl4eQLeXrXfNqmAlG3aFOXoIO2F258H4zEPtrM0fv17Mos6zcbdHo3cv
BZ21Gt/Dca4rYSB/Icn14OTW
-----END CERTIFICATE-----

2021-01-18T20:58:06.069912Z     info    sds     SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS" 

2021-01-18T20:58:06.070132Z     info    sds     Start SDS grpc server
2021-01-18T20:58:06.070190Z     error   invalid key, must be in form of regex pattern ^/app-health/[^\/]+/(livez|readyz)$
Error: invalid key, must be in form of regex pattern ^/app-health/[^\/]+/(livez|readyz)$
$ k logs jenkins-0 -n jenkins -c jenkins -f
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2021-01-18 20:59:19.634+0000 [id=1]     INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @585ms to org.eclipse.jetty.util.log.JavaUtilLog
2021-01-18 20:59:19.822+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file
2021-01-18 20:59:19.864+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2021-01-18 20:59:19.938+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-9.4.33.v20201020; built: 2020-10-20T23:39:24.803Z; git: 1be68755656cef678b79a2ef1c2ebbca99e25420; jvm 1.8.0_242-b08
2021-01-18 20:59:20.500+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2021-01-18 20:59:20.579+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2021-01-18 20:59:20.579+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2021-01-18 20:59:20.581+0000 [id=1]     INFO    o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 660000ms
2021-01-18 20:59:21.101+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2021-01-18 20:59:21.267+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@18317edc{Jenkins v2.263.2,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
2021-01-18 20:59:21.298+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@6be46e8f{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2021-01-18 20:59:21.299+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: Started @2250ms
2021-01-18 20:59:21.303+0000 [id=21]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2021-01-18 20:59:22.879+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
2021-01-18 20:59:24.571+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2021-01-18 20:59:35.420+0000 [id=26]    WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class hudson.plugins.disk_usage.DiskUsageProjectActionFactory$DescriptorImpl$1 in file:/var/jenkins_home/plugins/disk-usage/WEB-INF/lib/disk-usage.jar; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
2021-01-18 20:59:35.453+0000 [id=26]    WARNING o.j.r.u.AnonymousClassWarnings#warn: Attempt to (de-)serialize anonymous class hudson.plugins.disk_usage.DiskUsageOvearallGraphGenerator$DiskUsageRecord$1 in file:/var/jenkins_home/plugins/disk-usage/WEB-INF/lib/disk-usage.jar; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
2021-01-18 20:59:36.732+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2021-01-18 20:59:36.747+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
2021-01-18 20:59:39.580+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2021-01-18 20:59:39.800+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
2021-01-18 20:59:40.605+0000 [id=29]    INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted
2021-01-18 20:59:42.305+0000 [id=28]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2021-01-18 20:59:42.307+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2021-01-18 20:59:42.480+0000 [id=46]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
2021-01-18 20:59:42.555+0000 [id=46]    INFO    hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2021-01-18 20:59:42.781+0000 [id=46]    INFO    hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at hudson.model.DownloadService.loadJSON(DownloadService.java:116)
        at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:216)
        at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:211)
        at hudson.PluginManager.checkUpdatesServer(PluginManager.java:1768)
        at hudson.util.Retrier.start(Retrier.java:63)
        at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:1739)
        at jenkins.DailyCheck.execute(DailyCheck.java:93)
        at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
        at java.lang.Thread.run(Thread.java:748)
2021-01-18 20:59:42.784+0000 [id=46]    INFO    hudson.util.Retrier#start: Calling the listener of the allowed exception 'Connection refused (Connection refused)' at the attempt #1 to do the action check updates server
2021-01-18 20:59:42.788+0000 [id=46]    INFO    hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2021-01-18 20:59:42.789+0000 [id=46]    SEVERE  hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: ConnectException: Connection refused (Connection refused)
2021-01-18 20:59:42.799+0000 [id=46]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 311 ms
2021-01-18 20:59:43.128+0000 [id=27]    INFO    o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@2dc6c768: display name [Root WebApplicationContext]; startup date [Mon Jan 18 20:59:43 UTC 2021]; root of context hierarchy
2021-01-18 20:59:43.129+0000 [id=27]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@2dc6c768]: org.springframework.beans.factory.support.DefaultListableBeanFactory@3e5a7be5
2021-01-18 20:59:43.154+0000 [id=27]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3e5a7be5: defining beans [authenticationManager]; root of factory hierarchy
2021-01-18 20:59:43.364+0000 [id=27]    INFO    o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@1471e38c: display name [Root WebApplicationContext]; startup date [Mon Jan 18 20:59:43 UTC 2021]; root of context hierarchy
2021-01-18 20:59:43.364+0000 [id=27]    INFO    o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@1471e38c]: org.springframework.beans.factory.support.DefaultListableBeanFactory@70095554
2021-01-18 20:59:43.365+0000 [id=27]    INFO    o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@70095554: defining beans [filter,legacy]; root of factory hierarchy
2021-01-18 20:59:43.470+0000 [id=27]    INFO    o.c.j.p.k.KubernetesClientProvider$SaveableListenerImpl#onChange: Invalidating Kubernetes client: kubernetes null
2021-01-18 20:59:43.490+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2021-01-18 20:59:43.534+0000 [id=20]    INFO    o.c.j.p.k.KubernetesClientProvider$SaveableListenerImpl#onChange: Invalidating Kubernetes client: kubernetes null
2021-01-18 20:59:43.630+0000 [id=20]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running
$ k logs jenkins-0 -n jenkins -c config-reload -f
/usr/local/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.0) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
[2021-01-18 20:55:08] Starting collector
[2021-01-18 20:55:08] No folder annotation was provided, defaulting to k8s-sidecar-target-directory
[2021-01-18 20:55:08] Selected resource type: ('configmap',)
[2021-01-18 20:55:08] Config for cluster api loaded...
[2021-01-18 20:55:08] Unique filenames will not be enforced.
[2021-01-18 20:55:13] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88474b2f50>: Failed to establish a new connection: [Errno 111] Connection refused'))

[2021-01-18 20:55:18] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88474c4f10>: Failed to establish a new connection: [Errno 111] Connection refused'))

[2021-01-18 20:55:23] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88474d2ed0>: Failed to establish a new connection: [Errno 111] Connection refused'))

[2021-01-18 20:55:28] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8847465f10>: Failed to establish a new connection: [Errno 111] Connection refused'))

[2021-01-18 20:55:33] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88474d2f90>: Failed to establish a new connection: [Errno 111] Connection refused'))

[2021-01-18 20:55:38] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88474ddf50>: Failed to establish a new connection: [Errno 111] Connection refused'))

[2021-01-18 20:55:43] MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='172.20.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/jenkins/configmaps?labelSelector=jenkins-jenkins-config&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f88474ddcd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
$ k logs jenkins-0 -n jenkins -c istio-init -f
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1611003306251889857.txt305723092
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1611003306251889857.txt305723092
Writing following contents to rules file:  /tmp/ip6tables-rules-1611003306535528582.txt507517475

ip6tables-restore --noflush /tmp/ip6tables-rules-1611003306535528582.txt507517475
iptables-save 
# Generated by iptables-save v1.6.1 on Mon Jan 18 20:55:06 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Mon Jan 18 20:55:06 2021

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

hasakura12 commented 3 years ago

The error in istio-proxy sidecar container 

error   invalid key, must be in form of regex pattern ^/app-health/[^\/]+/(livez|readyz)$
Error: invalid key, must be in form of regex pattern ^/app-health/[^\/]+/(livez|readyz)$

was caused by istio 1.7.0's issue (istio-proxy doesn't allow rewritten startup probe).

After upgrading istioctl version to 1.7.6, the issue disappeared

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue is being automatically closed due to inactivity.