search

kyma-project / test-infra

Test infrastructure for the Kyma project.
https://status.build.kyma-project.io/
Apache License 2.0
38 stars 181 forks source link

prow: Sidecar sometimes gets terminated, resulting in incorrect job status #8473

Closed Ressetkk closed 12 months ago

Ressetkk commented 1 year ago

/kind flake /area ci /help

Problem

This is the meta Issue for figuring out what's wrong. Sidecar sometimes ends up in Terminated state despite the test container running fine. Affects usually K3d jobs for some reason. We need more investigation in that topic.

kdp ``` kdp e2331856-eb49-4c3c-9cec-b5400f7395ea Name: e2331856-eb49-4c3c-9cec-b5400f7395ea Namespace: default Priority: 0 Service Account: prowjob-default-sa Node: gke-untrusted-workload--standard-pool-14f257ae-shd7/10.109.140.8 Start Time: Tue, 01 Aug 2023 11:14:32 +0200 Labels: created-by-prow=true event-GUID=ccc81850-304b-11ee-9727-d880205805dc preset-dind-enabled=true preset-kind-volume-mounts=true prow.k8s.io/build-id=1686304363762421760 prow.k8s.io/context=pull-istio-integration prow.k8s.io/id=e2331856-eb49-4c3c-9cec-b5400f7395ea prow.k8s.io/is-optional=false prow.k8s.io/job=pull-istio-integration prow.k8s.io/plank-version=v20230714-24c9618c59 prow.k8s.io/pubsub.project=sap-kyma-prow prow.k8s.io/pubsub.runID=pull-istio-integration prow.k8s.io/pubsub.topic=prowjobs prow.k8s.io/refs.base_ref=main prow.k8s.io/refs.org=kyma-project prow.k8s.io/refs.pull=272 prow.k8s.io/refs.repo=istio prow.k8s.io/type=presubmit Annotations: cni.projectcalico.org/containerID: a4accd76d6be714f5306cd421724d17584cbd18ecf2af78d5acc51eba56cc8fa cni.projectcalico.org/podIP: 10.109.0.206/32 cni.projectcalico.org/podIPs: 10.109.0.206/32 description: Runs integration suite for istio operator kubernetes.io/limit-ranger: LimitRanger plugin set: memory request for container sidecar; cpu, memory limit for container sidecar; memory request for init container c... owner: goat prow.k8s.io/context: pull-istio-integration prow.k8s.io/job: pull-istio-integration Status: Running IP: 10.109.0.206 IPs: IP: 10.109.0.206 Init Containers: clonerefs: Container ID: containerd://29f2b9d20279b2b36df6990d1bff4731ba13282fac4e8cc4172206be146a62b5 Image: gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/clonerefs@sha256:2d0418c829d707824fe65add9534af2495aba033ca3bc05a84c1e42a2c4c190a Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:14:32 +0200 Finished: Tue, 01 Aug 2023 11:14:34 +0200 Ready: True Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: CLONEREFS_OPTIONS: {"src_root":"/home/prow/go","log":"/logs/clone.json","git_user_name":"ci-robot","git_user_email":"ci-robot@k8s.io","refs":[{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]}],"github_api_endpoints":["https://api.github.com"]} Mounts: /home/prow/go from code (rw) /logs from logs (rw) /tmp from clonerefs-tmp (rw) initupload: Container ID: containerd://bb5aa37d139186e274fab1154442d1db011292567cce4723c1e2d74fbed49253 Image: gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/initupload@sha256:0c3c12619bf7435cb23e6a0e65e72a7a2081364b445f94b5cf2cc4c8086a5471 Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:14:35 +0200 Finished: Tue, 01 Aug 2023 11:14:35 +0200 Ready: True Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: INITUPLOAD_OPTIONS: {"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false,"log":"/logs/clone.json"} JOB_SPEC: {"type":"presubmit","job":"pull-istio-integration","buildid":"1686304363762421760","prowjobid":"e2331856-eb49-4c3c-9cec-b5400f7395ea","refs":{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]},"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} Mounts: /logs from logs (rw) place-entrypoint: Container ID: containerd://38bb3856941edebd0f83f6df326aa8e1b748cb8e0731581a87848edf876842ab Image: gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/entrypoint@sha256:40912b7f0b186124bb5caca32fcf8692e7aa086250ac55945709f348f0f7d051 Port: Host Port: Args: --copy-mode-only State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:14:36 +0200 Finished: Tue, 01 Aug 2023 11:14:36 +0200 Ready: True Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: Mounts: /tools from tools (rw) Containers: test: Container ID: containerd://36b58aebf4c84629a8da46265b9b128e5f086c3bda94de20977803af4af18ec3 Image: europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d:v20230728-90a0fc69 Image ID: europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d@sha256:9a73e454bcdd18c2d1262f2f04f1d519de10579101687ecc170b679b7697f95f Port: Host Port: Command: /tools/entrypoint State: Running Started: Tue, 01 Aug 2023 11:14:37 +0200 Ready: True Restart Count: 0 Limits: cpu: 1500m memory: 2Gi Requests: cpu: 1 memory: 1Gi Environment: DOCKER_IN_DOCKER_ENABLED: true ARTIFACTS: /logs/artifacts BUILD_ID: 1686304363762421760 BUILD_NUMBER: 1686304363762421760 CI: true GOPATH: /home/prow/go JOB_NAME: pull-istio-integration JOB_SPEC: {"type":"presubmit","job":"pull-istio-integration","buildid":"1686304363762421760","prowjobid":"e2331856-eb49-4c3c-9cec-b5400f7395ea","refs":{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]},"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} JOB_TYPE: presubmit PROW_JOB_ID: e2331856-eb49-4c3c-9cec-b5400f7395ea PULL_BASE_REF: main PULL_BASE_SHA: c34608cfb02af12649258579030abae284a196d9 PULL_HEAD_REF: PULL_NUMBER: 272 PULL_PULL_SHA: a4706bbc9eb3be2d4e096bb307906c632a2e149b PULL_REFS: main:c34608cfb02af12649258579030abae284a196d9,272:a4706bbc9eb3be2d4e096bb307906c632a2e149b REPO_NAME: istio REPO_OWNER: kyma-project ENTRYPOINT_OPTIONS: {"timeout":10800000000000,"grace_period":600000000000,"artifact_dir":"/logs/artifacts","args":["/init.sh","bash","-c","set -e\nk3d registry create registry.localhost --port 5000\nk3d cluster create -a 0 --registry-use k3d-registry.localhost:5000 --servers-memory=10g --port 80:80@loadbalancer --port 443:443@loadbalancer --k3s-arg --disable=traefik@server:0\nIMG=\"k3d-registry.localhost:5000/istio-operator:latest\" make docker-build docker-push istio-integration-test\n"],"container_name":"test","process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"} Mounts: /docker-graph from docker-graph (rw) /home/prow/go from code (rw) /lib/modules from modules (ro) /logs from logs (rw) /sys/fs/cgroup from cgroup (rw) /tools from tools (rw) sidecar: Container ID: containerd://0ed6daffa7eb6534b3f48c37e1cc52c8401c972dc78413280c2b1ffd2ea5fdb5 Image: gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/sidecar@sha256:3a543aabcc032c34b015a0571f5ab0c0e1e2c02d2e1cfba657d28fe0c555b14b Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:14:37 +0200 Finished: Tue, 01 Aug 2023 11:14:37 +0200 Ready: False Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: JOB_SPEC: {"type":"presubmit","job":"pull-istio-integration","buildid":"1686304363762421760","prowjobid":"e2331856-eb49-4c3c-9cec-b5400f7395ea","refs":{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]},"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} SIDECAR_OPTIONS: {"gcs_options":{"items":["/logs/artifacts"],"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false},"entries":[{"args":["/init.sh","bash","-c","set -e\nk3d registry create registry.localhost --port 5000\nk3d cluster create -a 0 --registry-use k3d-registry.localhost:5000 --servers-memory=10g --port 80:80@loadbalancer --port 443:443@loadbalancer --k3s-arg --disable=traefik@server:0\nIMG=\"k3d-registry.localhost:5000/istio-operator:latest\" make docker-build docker-push istio-integration-test\n"],"container_name":"test","process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}],"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}} Mounts: /logs from logs (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: docker-graph: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: modules: Type: HostPath (bare host directory volume) Path: /lib/modules HostPathType: Directory cgroup: Type: HostPath (bare host directory volume) Path: /sys/fs/cgroup HostPathType: Directory logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: tools: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: clonerefs-tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: code: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 7m10s gke.io/optimize-utilization-scheduler Successfully assigned default/e2331856-eb49-4c3c-9cec-b5400f7395ea to gke-untrusted-workload--standard-pool-14f257ae-shd7 Normal Pulled 7m10s kubelet Container image "gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3" already present on machine Normal Created 7m10s kubelet Created container clonerefs Normal Started 7m10s kubelet Started container clonerefs Normal Pulled 7m7s kubelet Container image "gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3" already present on machine Normal Created 7m7s kubelet Created container initupload Normal Started 7m7s kubelet Started container initupload Normal Pulled 7m6s kubelet Container image "gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3" already present on machine Normal Created 7m6s kubelet Created container place-entrypoint Normal Started 7m6s kubelet Started container place-entrypoint Normal Pulled 7m5s kubelet Container image "europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d:v20230728-90a0fc69" already present on machine Normal Created 7m5s kubelet Created container test Normal Started 7m5s kubelet Started container test Normal Pulled 7m5s kubelet Container image "gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3" already present on machine Normal Created 7m5s kubelet Created container sidecar Normal Started 7m5s kubelet Started container sidecar ```
sidecar logs ``` {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:46","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"info","msg":"Starting to censor data","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","concurrency":2,"file":"k8s.io/test-infra/prow/sidecar/censor.go:56","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"debug","msg":"Censoring artifacts.","severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:79","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"debug","msg":"Loaded secrets to censor.","secrets":0,"severity":"debug","time":"2023-08-01T09:14:37Z"} {"buffer_size":3092,"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:90","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"debug","msg":"Determined censoring buffer size.","severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:140","func":"k8s.io/test-infra/prow/sidecar.Options.censor.func4","level":"debug","msg":"Censoring file.","path":"/logs/artifacts/dockerd.log","severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","duration":"491.08µs","file":"k8s.io/test-infra/prow/sidecar/censor.go:48","func":"k8s.io/test-infra/prow/sidecar.Options.censor.func1","level":"info","msg":"Finished censoring data","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/run.go:235","func":"k8s.io/test-infra/prow/sidecar.Options.doUpload","level":"info","msg":"Starting to upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:43","func":"k8s.io/test-infra/prow/gcsupload.Options.Run","level":"debug","msg":"Uploading to blob storage","options":{"items":["/logs/artifacts"],"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false},"severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:198","func":"k8s.io/test-infra/prow/gcsupload.gatherArtifacts","level":"info","msg":"Gathering artifacts from artifact directory: /logs/artifacts","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:216","func":"k8s.io/test-infra/prow/gcsupload.gatherArtifacts.func1","level":"info","msg":"Found /logs/artifacts/dockerd.log in artifact directory. Uploading as pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/artifacts/dockerd.log\n","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/1686304363762421760.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/artifacts/dockerd.log","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/artifacts/dockerd.log","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/1686304363762421760.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:79","func":"k8s.io/test-infra/prow/gcsupload.completeUpload","level":"info","msg":"Finished upload to blob storage","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/sidecar-logs.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/finished.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/build-log.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/finished.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/build-log.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/sidecar-logs.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:79","func":"k8s.io/test-infra/prow/gcsupload.completeUpload","level":"info","msg":"Finished upload to blob storage","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","duration":"341.53512ms","file":"k8s.io/test-infra/prow/sidecar/run.go:239","func":"k8s.io/test-infra/prow/sidecar.Options.doUpload.func1","level":"info","msg":"Finished uploading","severity":"info","time":"2023-08-01T09:14:37Z"} ```
kyma-bot commented 1 year ago

@Ressetkk: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kyma-project/test-infra/issues/8473): >/kind flake >/area ci >/help > >## Problem >This is the meta Issue for figuring out what's wrong. >Sidecar sometimes ends up in `Terminated` state despite the test container running fine. Affects usually K3d jobs for some reason. We need more investigation in that topic. > >
>kdp > >``` >kdp e2331856-eb49-4c3c-9cec-b5400f7395ea >Name: e2331856-eb49-4c3c-9cec-b5400f7395ea >Namespace: default >Priority: 0 >Service Account: prowjob-default-sa >Node: gke-untrusted-workload--standard-pool-14f257ae-shd7/10.109.140.8 >Start Time: Tue, 01 Aug 2023 11:14:32 +0200 >Labels: created-by-prow=true > event-GUID=ccc81850-304b-11ee-9727-d880205805dc > preset-dind-enabled=true > preset-kind-volume-mounts=true > prow.k8s.io/build-id=1686304363762421760 > prow.k8s.io/context=pull-istio-integration > prow.k8s.io/id=e2331856-eb49-4c3c-9cec-b5400f7395ea > prow.k8s.io/is-optional=false > prow.k8s.io/job=pull-istio-integration > prow.k8s.io/plank-version=v20230714-24c9618c59 > prow.k8s.io/pubsub.project=sap-kyma-prow > prow.k8s.io/pubsub.runID=pull-istio-integration > prow.k8s.io/pubsub.topic=prowjobs > prow.k8s.io/refs.base_ref=main > prow.k8s.io/refs.org=kyma-project > prow.k8s.io/refs.pull=272 > prow.k8s.io/refs.repo=istio > prow.k8s.io/type=presubmit >Annotations: cni.projectcalico.org/containerID: a4accd76d6be714f5306cd421724d17584cbd18ecf2af78d5acc51eba56cc8fa > cni.projectcalico.org/podIP: 10.109.0.206/32 > cni.projectcalico.org/podIPs: 10.109.0.206/32 > description: Runs integration suite for istio operator > kubernetes.io/limit-ranger: > LimitRanger plugin set: memory request for container sidecar; cpu, memory limit for container sidecar; memory request for init container c... > owner: goat > prow.k8s.io/context: pull-istio-integration > prow.k8s.io/job: pull-istio-integration >Status: Running >IP: 10.109.0.206 >IPs: > IP: 10.109.0.206 >Init Containers: > clonerefs: > Container ID: containerd://29f2b9d20279b2b36df6990d1bff4731ba13282fac4e8cc4172206be146a62b5 > Image: gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3 > Image ID: gcr.io/k8s-prow/clonerefs@sha256:2d0418c829d707824fe65add9534af2495aba033ca3bc05a84c1e42a2c4c190a > Port: > Host Port: > State: Terminated > Reason: Completed > Exit Code: 0 > Started: Tue, 01 Aug 2023 11:14:32 +0200 > Finished: Tue, 01 Aug 2023 11:14:34 +0200 > Ready: True > Restart Count: 0 > Limits: > cpu: 2500m > memory: 4Gi > Requests: > cpu: 100m > memory: 2Gi > Environment: > CLONEREFS_OPTIONS: {"src_root":"/home/prow/go","log":"/logs/clone.json","git_user_name":"ci-robot","git_user_email":"ci-robot@k8s.io","refs":[{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]}],"github_api_endpoints":["https://api.github.com"]} > Mounts: > /home/prow/go from code (rw) > /logs from logs (rw) > /tmp from clonerefs-tmp (rw) > initupload: > Container ID: containerd://bb5aa37d139186e274fab1154442d1db011292567cce4723c1e2d74fbed49253 > Image: gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3 > Image ID: gcr.io/k8s-prow/initupload@sha256:0c3c12619bf7435cb23e6a0e65e72a7a2081364b445f94b5cf2cc4c8086a5471 > Port: > Host Port: > State: Terminated > Reason: Completed > Exit Code: 0 > Started: Tue, 01 Aug 2023 11:14:35 +0200 > Finished: Tue, 01 Aug 2023 11:14:35 +0200 > Ready: True > Restart Count: 0 > Limits: > cpu: 2500m > memory: 4Gi > Requests: > cpu: 100m > memory: 2Gi > Environment: > INITUPLOAD_OPTIONS: {"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false,"log":"/logs/clone.json"} > JOB_SPEC: {"type":"presubmit","job":"pull-istio-integration","buildid":"1686304363762421760","prowjobid":"e2331856-eb49-4c3c-9cec-b5400f7395ea","refs":{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]},"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} > Mounts: > /logs from logs (rw) > place-entrypoint: > Container ID: containerd://38bb3856941edebd0f83f6df326aa8e1b748cb8e0731581a87848edf876842ab > Image: gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3 > Image ID: gcr.io/k8s-prow/entrypoint@sha256:40912b7f0b186124bb5caca32fcf8692e7aa086250ac55945709f348f0f7d051 > Port: > Host Port: > Args: > --copy-mode-only > State: Terminated > Reason: Completed > Exit Code: 0 > Started: Tue, 01 Aug 2023 11:14:36 +0200 > Finished: Tue, 01 Aug 2023 11:14:36 +0200 > Ready: True > Restart Count: 0 > Limits: > cpu: 2500m > memory: 4Gi > Requests: > cpu: 100m > memory: 2Gi > Environment: > Mounts: > /tools from tools (rw) >Containers: > test: > Container ID: containerd://36b58aebf4c84629a8da46265b9b128e5f086c3bda94de20977803af4af18ec3 > Image: europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d:v20230728-90a0fc69 > Image ID: europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d@sha256:9a73e454bcdd18c2d1262f2f04f1d519de10579101687ecc170b679b7697f95f > Port: > Host Port: > Command: > /tools/entrypoint > State: Running > Started: Tue, 01 Aug 2023 11:14:37 +0200 > Ready: True > Restart Count: 0 > Limits: > cpu: 1500m > memory: 2Gi > Requests: > cpu: 1 > memory: 1Gi > Environment: > DOCKER_IN_DOCKER_ENABLED: true > ARTIFACTS: /logs/artifacts > BUILD_ID: 1686304363762421760 > BUILD_NUMBER: 1686304363762421760 > CI: true > GOPATH: /home/prow/go > JOB_NAME: pull-istio-integration > JOB_SPEC: {"type":"presubmit","job":"pull-istio-integration","buildid":"1686304363762421760","prowjobid":"e2331856-eb49-4c3c-9cec-b5400f7395ea","refs":{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]},"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} > JOB_TYPE: presubmit > PROW_JOB_ID: e2331856-eb49-4c3c-9cec-b5400f7395ea > PULL_BASE_REF: main > PULL_BASE_SHA: c34608cfb02af12649258579030abae284a196d9 > PULL_HEAD_REF: > PULL_NUMBER: 272 > PULL_PULL_SHA: a4706bbc9eb3be2d4e096bb307906c632a2e149b > PULL_REFS: main:c34608cfb02af12649258579030abae284a196d9,272:a4706bbc9eb3be2d4e096bb307906c632a2e149b > REPO_NAME: istio > REPO_OWNER: kyma-project > ENTRYPOINT_OPTIONS: {"timeout":10800000000000,"grace_period":600000000000,"artifact_dir":"/logs/artifacts","args":["/init.sh","bash","-c","set -e\nk3d registry create registry.localhost --port 5000\nk3d cluster create -a 0 --registry-use k3d-registry.localhost:5000 --servers-memory=10g --port 80:80@loadbalancer --port 443:443@loadbalancer --k3s-arg --disable=traefik@server:0\nIMG=\"k3d-registry.localhost:5000/istio-operator:latest\" make docker-build docker-push istio-integration-test\n"],"container_name":"test","process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"} > Mounts: > /docker-graph from docker-graph (rw) > /home/prow/go from code (rw) > /lib/modules from modules (ro) > /logs from logs (rw) > /sys/fs/cgroup from cgroup (rw) > /tools from tools (rw) > sidecar: > Container ID: containerd://0ed6daffa7eb6534b3f48c37e1cc52c8401c972dc78413280c2b1ffd2ea5fdb5 > Image: gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3 > Image ID: gcr.io/k8s-prow/sidecar@sha256:3a543aabcc032c34b015a0571f5ab0c0e1e2c02d2e1cfba657d28fe0c555b14b > Port: > Host Port: > State: Terminated > Reason: Completed > Exit Code: 0 > Started: Tue, 01 Aug 2023 11:14:37 +0200 > Finished: Tue, 01 Aug 2023 11:14:37 +0200 > Ready: False > Restart Count: 0 > Limits: > cpu: 2500m > memory: 4Gi > Requests: > cpu: 100m > memory: 2Gi > Environment: > JOB_SPEC: {"type":"presubmit","job":"pull-istio-integration","buildid":"1686304363762421760","prowjobid":"e2331856-eb49-4c3c-9cec-b5400f7395ea","refs":{"org":"kyma-project","repo":"istio","repo_link":"https://github.com/kyma-project/istio","base_ref":"main","base_sha":"c34608cfb02af12649258579030abae284a196d9","base_link":"https://github.com/kyma-project/istio/commit/c34608cfb02af12649258579030abae284a196d9","pulls":[{"number":272,"author":"triffer","sha":"a4706bbc9eb3be2d4e096bb307906c632a2e149b","title":"Istio resources reconciler","link":"https://github.com/kyma-project/istio/pull/272","commit_link":"https://github.com/kyma-project/istio/pull/272/commits/a4706bbc9eb3be2d4e096bb307906c632a2e149b","author_link":"https://github.com/triffer"}]},"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} > SIDECAR_OPTIONS: {"gcs_options":{"items":["/logs/artifacts"],"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false},"entries":[{"args":["/init.sh","bash","-c","set -e\nk3d registry create registry.localhost --port 5000\nk3d cluster create -a 0 --registry-use k3d-registry.localhost:5000 --servers-memory=10g --port 80:80@loadbalancer --port 443:443@loadbalancer --k3s-arg --disable=traefik@server:0\nIMG=\"k3d-registry.localhost:5000/istio-operator:latest\" make docker-build docker-push istio-integration-test\n"],"container_name":"test","process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}],"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}} > Mounts: > /logs from logs (rw) >Conditions: > Type Status > Initialized True > Ready False > ContainersReady False > PodScheduled True >Volumes: > docker-graph: > Type: EmptyDir (a temporary directory that shares a pod's lifetime) > Medium: > SizeLimit: > modules: > Type: HostPath (bare host directory volume) > Path: /lib/modules > HostPathType: Directory > cgroup: > Type: HostPath (bare host directory volume) > Path: /sys/fs/cgroup > HostPathType: Directory > logs: > Type: EmptyDir (a temporary directory that shares a pod's lifetime) > Medium: > SizeLimit: > tools: > Type: EmptyDir (a temporary directory that shares a pod's lifetime) > Medium: > SizeLimit: > clonerefs-tmp: > Type: EmptyDir (a temporary directory that shares a pod's lifetime) > Medium: > SizeLimit: > code: > Type: EmptyDir (a temporary directory that shares a pod's lifetime) > Medium: > SizeLimit: >QoS Class: Burstable >Node-Selectors: >Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s > node.kubernetes.io/unreachable:NoExecute op=Exists for 300s >Events: > Type Reason Age From Message > ---- ------ ---- ---- ------- > Normal Scheduled 7m10s gke.io/optimize-utilization-scheduler Successfully assigned default/e2331856-eb49-4c3c-9cec-b5400f7395ea to gke-untrusted-workload--standard-pool-14f257ae-shd7 > Normal Pulled 7m10s kubelet Container image "gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3" already present on machine > Normal Created 7m10s kubelet Created container clonerefs > Normal Started 7m10s kubelet Started container clonerefs > Normal Pulled 7m7s kubelet Container image "gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3" already present on machine > Normal Created 7m7s kubelet Created container initupload > Normal Started 7m7s kubelet Started container initupload > Normal Pulled 7m6s kubelet Container image "gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3" already present on machine > Normal Created 7m6s kubelet Created container place-entrypoint > Normal Started 7m6s kubelet Started container place-entrypoint > Normal Pulled 7m5s kubelet Container image "europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d:v20230728-90a0fc69" already present on machine > Normal Created 7m5s kubelet Created container test > Normal Started 7m5s kubelet Started container test > Normal Pulled 7m5s kubelet Container image "gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3" already present on machine > Normal Created 7m5s kubelet Created container sidecar > Normal Started 7m5s kubelet Started container sidecar >``` > >
sidecar logs ``` {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:46","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"info","msg":"Starting to censor data","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","concurrency":2,"file":"k8s.io/test-infra/prow/sidecar/censor.go:56","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"debug","msg":"Censoring artifacts.","severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:79","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"debug","msg":"Loaded secrets to censor.","secrets":0,"severity":"debug","time":"2023-08-01T09:14:37Z"} {"buffer_size":3092,"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:90","func":"k8s.io/test-infra/prow/sidecar.Options.censor","level":"debug","msg":"Determined censoring buffer size.","severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/censor.go:140","func":"k8s.io/test-infra/prow/sidecar.Options.censor.func4","level":"debug","msg":"Censoring file.","path":"/logs/artifacts/dockerd.log","severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","duration":"491.08µs","file":"k8s.io/test-infra/prow/sidecar/censor.go:48","func":"k8s.io/test-infra/prow/sidecar.Options.censor.func1","level":"info","msg":"Finished censoring data","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/sidecar/run.go:235","func":"k8s.io/test-infra/prow/sidecar.Options.doUpload","level":"info","msg":"Starting to upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:43","func":"k8s.io/test-infra/prow/gcsupload.Options.Run","level":"debug","msg":"Uploading to blob storage","options":{"items":["/logs/artifacts"],"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false},"severity":"debug","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:198","func":"k8s.io/test-infra/prow/gcsupload.gatherArtifacts","level":"info","msg":"Gathering artifacts from artifact directory: /logs/artifacts","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:216","func":"k8s.io/test-infra/prow/gcsupload.gatherArtifacts.func1","level":"info","msg":"Found /logs/artifacts/dockerd.log in artifact directory. Uploading as pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/artifacts/dockerd.log\n","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/1686304363762421760.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/artifacts/dockerd.log","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/artifacts/dockerd.log","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/directory/pull-istio-integration/1686304363762421760.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/latest-build.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:79","func":"k8s.io/test-infra/prow/gcsupload.completeUpload","level":"info","msg":"Finished upload to blob storage","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/sidecar-logs.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/finished.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/build-log.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:89","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/finished.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/build-log.txt","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","dest":"gs://kyma-prow-logs/pr-logs/pull/kyma-project_istio/272/pull-istio-integration/1686304363762421760/sidecar-logs.json","file":"k8s.io/test-infra/prow/pod-utils/gcs/upload.go:117","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","file":"k8s.io/test-infra/prow/gcsupload/run.go:79","func":"k8s.io/test-infra/prow/gcsupload.completeUpload","level":"info","msg":"Finished upload to blob storage","severity":"info","time":"2023-08-01T09:14:37Z"} {"component":"sidecar","duration":"341.53512ms","file":"k8s.io/test-infra/prow/sidecar/run.go:239","func":"k8s.io/test-infra/prow/sidecar.Options.doUpload.func1","level":"info","msg":"Finished uploading","severity":"info","time":"2023-08-01T09:14:37Z"} ```

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Ressetkk commented 1 year ago
``` Name: e0b0e325-f8dc-4880-a934-8a4bf53ec8c8 Namespace: default Priority: 0 Service Account: prowjob-default-sa Node: gke-untrusted-workload--standard-pool-14f257ae-shd7/10.109.140.8 Start Time: Tue, 01 Aug 2023 11:49:43 +0200 Labels: created-by-prow=true event-GUID=af8ec3b0-3050-11ee-8b5b-c0fdd718a111 preset-build-pr=true preset-dind-enabled=true preset-gc-project-env=true preset-kind-volume-mounts=true preset-kyma-guard-bot-github-token=true preset-sa-vm-kyma-integration=true prow.k8s.io/build-id=1686313196878565376 prow.k8s.io/context=pre-main-compass-integration-with-dump prow.k8s.io/id=e0b0e325-f8dc-4880-a934-8a4bf53ec8c8 prow.k8s.io/is-optional=false prow.k8s.io/job=pre-main-compass-integration-with-dump prow.k8s.io/plank-version=v20230714-24c9618c59 prow.k8s.io/pubsub.project=sap-kyma-prow prow.k8s.io/pubsub.runID=pre-main-compass-integration-with-dump prow.k8s.io/pubsub.topic=prowjobs prow.k8s.io/refs.base_ref=main prow.k8s.io/refs.org=kyma-incubator prow.k8s.io/refs.pull=3186 prow.k8s.io/refs.repo=compass prow.k8s.io/type=presubmit Annotations: cni.projectcalico.org/containerID: de644fe7aa3e66de663f0fe6cba1499c0d6e5d7064d32cc4cd298b8ae6dd2911 cni.projectcalico.org/podIP: 10.109.0.227/32 cni.projectcalico.org/podIPs: 10.109.0.227/32 description: Compass Integration Tests with DB dump kubernetes.io/limit-ranger: LimitRanger plugin set: memory request for container sidecar; cpu, memory limit for container sidecar; memory request for init container c... owner: compass prow.k8s.io/context: pre-main-compass-integration-with-dump prow.k8s.io/job: pre-main-compass-integration-with-dump Status: Running IP: 10.109.0.227 IPs: IP: 10.109.0.227 Init Containers: clonerefs: Container ID: containerd://46a9c0021cc82aaafa5b82f63e52081639feb90e5a71b452de871c05fb2d1e0b Image: gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/clonerefs@sha256:2d0418c829d707824fe65add9534af2495aba033ca3bc05a84c1e42a2c4c190a Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:49:44 +0200 Finished: Tue, 01 Aug 2023 11:49:59 +0200 Ready: True Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: CLONEREFS_OPTIONS: {"src_root":"/home/prow/go","log":"/logs/clone.json","git_user_name":"ci-robot","git_user_email":"ci-robot@k8s.io","refs":[{"org":"kyma-incubator","repo":"compass","repo_link":"https://github.com/kyma-incubator/compass","base_ref":"main","base_sha":"8708ae8d9f28b23c18a4cc0eeea4ec030c634632","base_link":"https://github.com/kyma-incubator/compass/commit/8708ae8d9f28b23c18a4cc0eeea4ec030c634632","pulls":[{"number":3186,"author":"ognyvrac","sha":"9fffd2e17013b000dab9e47dba2f2a0da2af9412","title":"[WIP] Add own Ory installation","link":"https://github.com/kyma-incubator/compass/pull/3186","commit_link":"https://github.com/kyma-incubator/compass/pull/3186/commits/9fffd2e17013b000dab9e47dba2f2a0da2af9412","author_link":"https://github.com/ognyvrac"}]},{"org":"kyma-project","repo":"test-infra","base_ref":"main"}],"github_api_endpoints":["https://api.github.com"]} Mounts: /home/prow/go from code (rw) /logs from logs (rw) /tmp from clonerefs-tmp (rw) initupload: Container ID: containerd://9e52c6177c907114c813423c9f8937bac75c2803423f8ebb401359bb8a41b209 Image: gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/initupload@sha256:0c3c12619bf7435cb23e6a0e65e72a7a2081364b445f94b5cf2cc4c8086a5471 Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:49:59 +0200 Finished: Tue, 01 Aug 2023 11:50:00 +0200 Ready: True Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: INITUPLOAD_OPTIONS: {"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false,"log":"/logs/clone.json"} JOB_SPEC: {"type":"presubmit","job":"pre-main-compass-integration-with-dump","buildid":"1686313196878565376","prowjobid":"e0b0e325-f8dc-4880-a934-8a4bf53ec8c8","refs":{"org":"kyma-incubator","repo":"compass","repo_link":"https://github.com/kyma-incubator/compass","base_ref":"main","base_sha":"8708ae8d9f28b23c18a4cc0eeea4ec030c634632","base_link":"https://github.com/kyma-incubator/compass/commit/8708ae8d9f28b23c18a4cc0eeea4ec030c634632","pulls":[{"number":3186,"author":"ognyvrac","sha":"9fffd2e17013b000dab9e47dba2f2a0da2af9412","title":"[WIP] Add own Ory installation","link":"https://github.com/kyma-incubator/compass/pull/3186","commit_link":"https://github.com/kyma-incubator/compass/pull/3186/commits/9fffd2e17013b000dab9e47dba2f2a0da2af9412","author_link":"https://github.com/ognyvrac"}]},"extra_refs":[{"org":"kyma-project","repo":"test-infra","base_ref":"main"}],"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} Mounts: /logs from logs (rw) place-entrypoint: Container ID: containerd://9e72f3b398ed278e1560e3b50483d34bb9a1855d673f268d9ad9cd2376b55820 Image: gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/entrypoint@sha256:40912b7f0b186124bb5caca32fcf8692e7aa086250ac55945709f348f0f7d051 Port: Host Port: Args: --copy-mode-only State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:50:01 +0200 Finished: Tue, 01 Aug 2023 11:50:01 +0200 Ready: True Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: Mounts: /tools from tools (rw) Containers: test: Container ID: containerd://51a21d05f98ef9702ae0c81ce6523cc17e6238a34f957b975d8a734bbba9ef0a Image: europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d:v20230728-90a0fc69 Image ID: europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d@sha256:9a73e454bcdd18c2d1262f2f04f1d519de10579101687ecc170b679b7697f95f Port: Host Port: Command: /tools/entrypoint State: Running Started: Tue, 01 Aug 2023 11:50:02 +0200 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 3Gi Requests: cpu: 1 memory: 2Gi Environment: GO111MODULE: on KYMA_PROJECT_DIR: /home/prow/go/src/github.com/kyma-project CLOUDSDK_CORE_PROJECT: sap-kyma-prow-workloads GOOGLE_APPLICATION_CREDENTIALS: /etc/credentials/sa-vm-kyma-integration/service-account.json DOCKER_IN_DOCKER_ENABLED: true BUILD_TYPE: pr DOCKER_PUSH_DIRECTORY: /pr BOT_GITHUB_TOKEN: Optional: false ARTIFACTS: /logs/artifacts BUILD_ID: 1686313196878565376 BUILD_NUMBER: 1686313196878565376 CI: true GOPATH: /home/prow/go JOB_NAME: pre-main-compass-integration-with-dump JOB_SPEC: {"type":"presubmit","job":"pre-main-compass-integration-with-dump","buildid":"1686313196878565376","prowjobid":"e0b0e325-f8dc-4880-a934-8a4bf53ec8c8","refs":{"org":"kyma-incubator","repo":"compass","repo_link":"https://github.com/kyma-incubator/compass","base_ref":"main","base_sha":"8708ae8d9f28b23c18a4cc0eeea4ec030c634632","base_link":"https://github.com/kyma-incubator/compass/commit/8708ae8d9f28b23c18a4cc0eeea4ec030c634632","pulls":[{"number":3186,"author":"ognyvrac","sha":"9fffd2e17013b000dab9e47dba2f2a0da2af9412","title":"[WIP] Add own Ory installation","link":"https://github.com/kyma-incubator/compass/pull/3186","commit_link":"https://github.com/kyma-incubator/compass/pull/3186/commits/9fffd2e17013b000dab9e47dba2f2a0da2af9412","author_link":"https://github.com/ognyvrac"}]},"extra_refs":[{"org":"kyma-project","repo":"test-infra","base_ref":"main"}],"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} JOB_TYPE: presubmit PROW_JOB_ID: e0b0e325-f8dc-4880-a934-8a4bf53ec8c8 PULL_BASE_REF: main PULL_BASE_SHA: 8708ae8d9f28b23c18a4cc0eeea4ec030c634632 PULL_HEAD_REF: PULL_NUMBER: 3186 PULL_PULL_SHA: 9fffd2e17013b000dab9e47dba2f2a0da2af9412 PULL_REFS: main:8708ae8d9f28b23c18a4cc0eeea4ec030c634632,3186:9fffd2e17013b000dab9e47dba2f2a0da2af9412 REPO_NAME: compass REPO_OWNER: kyma-incubator ENTRYPOINT_OPTIONS: {"timeout":10800000000000,"grace_period":600000000000,"artifact_dir":"/logs/artifacts","args":["/init.sh","installation/scripts/prow/jobs/provision-compass.sh","--dump-db"],"container_name":"test","process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"} Mounts: /docker-graph from docker-graph (rw) /etc/credentials/sa-vm-kyma-integration from sa-vm-kyma-integration (ro) /etc/github from kyma-guard-bot-token-path (ro) /home/prow/go from code (rw) /lib/modules from modules (ro) /logs from logs (rw) /root/.ssh/user from sa-vm-kyma-ssh-key (rw) /sys/fs/cgroup from cgroup (rw) /tools from tools (rw) sidecar: Container ID: containerd://a1d6a999a41d34ce25d97a9024ef59f9d616b786fd48f79b7fbc872e07279f1d Image: gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3 Image ID: gcr.io/k8s-prow/sidecar@sha256:3a543aabcc032c34b015a0571f5ab0c0e1e2c02d2e1cfba657d28fe0c555b14b Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 01 Aug 2023 11:50:03 +0200 Finished: Tue, 01 Aug 2023 11:50:04 +0200 Ready: False Restart Count: 0 Limits: cpu: 2500m memory: 4Gi Requests: cpu: 100m memory: 2Gi Environment: JOB_SPEC: {"type":"presubmit","job":"pre-main-compass-integration-with-dump","buildid":"1686313196878565376","prowjobid":"e0b0e325-f8dc-4880-a934-8a4bf53ec8c8","refs":{"org":"kyma-incubator","repo":"compass","repo_link":"https://github.com/kyma-incubator/compass","base_ref":"main","base_sha":"8708ae8d9f28b23c18a4cc0eeea4ec030c634632","base_link":"https://github.com/kyma-incubator/compass/commit/8708ae8d9f28b23c18a4cc0eeea4ec030c634632","pulls":[{"number":3186,"author":"ognyvrac","sha":"9fffd2e17013b000dab9e47dba2f2a0da2af9412","title":"[WIP] Add own Ory installation","link":"https://github.com/kyma-incubator/compass/pull/3186","commit_link":"https://github.com/kyma-incubator/compass/pull/3186/commits/9fffd2e17013b000dab9e47dba2f2a0da2af9412","author_link":"https://github.com/ognyvrac"}]},"extra_refs":[{"org":"kyma-project","repo":"test-infra","base_ref":"main"}],"decoration_config":{"timeout":"3h0m0s","grace_period":"10m0s","utility_images":{"clonerefs":"gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3","initupload":"gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3","entrypoint":"gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3","sidecar":"gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3"},"gcs_configuration":{"bucket":"kyma-prow-logs","path_strategy":"explicit"},"gcs_credentials_secret":"","default_service_account_name":"prowjob-default-sa","censor_secrets":true,"censoring_options":{"censoring_concurrency":2,"censoring_buffer_size":3092}}} SIDECAR_OPTIONS: {"gcs_options":{"items":["/logs/artifacts"],"bucket":"kyma-prow-logs","path_strategy":"explicit","dry_run":false},"entries":[{"args":["/init.sh","installation/scripts/prow/jobs/provision-compass.sh","--dump-db"],"container_name":"test","process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}],"censoring_options":{"secret_directories":["/etc/credentials/sa-vm-kyma-integration","/root/.ssh/user","/etc/github"],"censoring_concurrency":2,"censoring_buffer_size":3092}} Mounts: /etc/credentials/sa-vm-kyma-integration from sa-vm-kyma-integration (ro) /etc/github from kyma-guard-bot-token-path (ro) /logs from logs (rw) /root/.ssh/user from sa-vm-kyma-ssh-key (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: sa-vm-kyma-integration: Type: Secret (a volume populated by a Secret) SecretName: sa-vm-kyma-integration Optional: false sa-vm-kyma-ssh-key: Type: Secret (a volume populated by a Secret) SecretName: sa-vm-kyma-ssh-key Optional: false docker-graph: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kyma-guard-bot-token-path: Type: Secret (a volume populated by a Secret) SecretName: kyma-guard-bot-github-token Optional: false modules: Type: HostPath (bare host directory volume) Path: /lib/modules HostPathType: Directory cgroup: Type: HostPath (bare host directory volume) Path: /sys/fs/cgroup HostPathType: Directory logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: tools: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: clonerefs-tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: code: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m7s gke.io/optimize-utilization-scheduler Successfully assigned default/e0b0e325-f8dc-4880-a934-8a4bf53ec8c8 to gke-untrusted-workload--standard-pool-14f257ae-shd7 Normal Pulled 5m6s kubelet Container image "gcr.io/k8s-prow/clonerefs:v20230728-f11c40f6c3" already present on machine Normal Created 5m6s kubelet Created container clonerefs Normal Started 5m6s kubelet Started container clonerefs Normal Pulled 4m51s kubelet Container image "gcr.io/k8s-prow/initupload:v20230728-f11c40f6c3" already present on machine Normal Created 4m51s kubelet Created container initupload Normal Started 4m51s kubelet Started container initupload Normal Pulled 4m49s kubelet Container image "gcr.io/k8s-prow/entrypoint:v20230728-f11c40f6c3" already present on machine Normal Created 4m49s kubelet Created container place-entrypoint Normal Started 4m49s kubelet Started container place-entrypoint Normal Pulled 4m48s kubelet Container image "europe-docker.pkg.dev/kyma-project/prod/testimages/e2e-dind-k3d:v20230728-90a0fc69" already present on machine Normal Created 4m48s kubelet Created container test Normal Started 4m48s kubelet Started container test Normal Pulled 4m48s kubelet Container image "gcr.io/k8s-prow/sidecar:v20230728-f11c40f6c3" already present on machine Normal Created 4m48s kubelet Created container sidecar Normal Started 4m47s kubelet Started container sidecar ```
chrkl commented 1 year ago

Here a few job that probably failed because of this issue: https://status.build.kyma-project.io/view/gs/kyma-prow-logs/pr-logs/pull/kyma-project_telemetry-manager/353/pull-telemetry-manager-e2e-test-logging/1693979907152941056 https://status.build.kyma-project.io/view/gs/kyma-prow-logs/pr-logs/pull/kyma-project_telemetry-manager/353/pull-telemetry-manager-e2e-test-logging-release/1693979907069054976 https://status.build.kyma-project.io/view/gs/kyma-prow-logs/pr-logs/pull/kyma-project_telemetry-manager/353/pull-telemetry-manager-e2e-test-metrics/1693979907266187264 https://status.build.kyma-project.io/view/gs/kyma-prow-logs/pr-logs/pull/kyma-project_telemetry-manager/353/pull-telemetry-manager-integration-test-istio/1693979924366364672

Ressetkk commented 1 year ago

the job is working, but because sidecar can't watch file due to the error regarding inotify watches no space left on the device it cannot watch container logs, and gets terminated with failure state, resulting in an failure

Added logic that cleans up all started containers in docker daemon after execution. let's hope it will resolve the problem

halamix2 commented 1 year ago

I've ran this on one of the node VMs

# sysctl fs.inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 12288

# find /proc/[0-9]*/fdinfo -type f 2>/dev/null | xargs grep ^inotify 2>/dev/null | wc -l
12264

kubelet is consuming almost all of these inotify watchers (over 11700)

Ressetkk commented 1 year ago

As a workaround we've added small DaemonSet that updates inotify max_user_watches to mitigate this error. We are going to monitor this further.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"DaemonSet","metadata":{"annotations":{},"labels":{"app":"sysctl-ds","version":"v1"},"name":"sysctl-ds","namespace":"default"},"spec":{"selector":{"matchLabels":{"name":"sysctl-ds"}},"template":{"metadata":{"labels":{"name":"sysctl-ds"}},"spec":{"containers":[{"command":["/bin/sh","-c","while true; do\n  sysctl -w fs.inotify.max_user_watches=524288\n  sleep 10\ndone\n"],"image":"alpine:3.18.3","imagePullPolicy":"IfNotPresent","name":"sysctl-buddy","securityContext":{"privileged":true}}]}},"updateStrategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}}}
  creationTimestamp: "2023-09-25T09:05:49Z"
  generation: 1
  labels:
    app: sysctl-ds
    version: v1
  name: sysctl-ds
  namespace: default
  resourceVersion: "645578750"
  uid: 7a923d5c-d1ab-4759-8697-d99ad28267cf
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: sysctl-ds
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: sysctl-ds
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - |
          while true; do
            sysctl -w fs.inotify.max_user_watches=524288
            sleep 10
          done
        image: alpine:3.18.3
        imagePullPolicy: IfNotPresent
        name: sysctl-buddy
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 4
  desiredNumberScheduled: 4
  numberAvailable: 4
  numberMisscheduled: 0
  numberReady: 4
  observedGeneration: 1
  updatedNumberScheduled: 4
Ressetkk commented 12 months ago

After applying this daemonset problem hasn't occurred ever since. /close

kyma-bot commented 12 months ago

@Ressetkk: Closing this issue.

In response to [this](https://github.com/kyma-project/test-infra/issues/8473#issuecomment-1782737688): >After applying this daemonset problem hasn't occurred ever since. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.