Closed hgye closed 2 years ago
workspace failed to startup
using helm to install gitpod self-hosted version v0.10.0, cluster node info
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master.my-company.loc Ready control-plane,master 136d v1.20.7 172.16.10.11 <none> CentOS Linux 7 (Core) 3.10.0-1160.24.1.el7.x86_64 containerd://1.4.4 node1.my-company.loc Ready <none> 136d v1.20.7 172.16.10.12 <none> CentOS Linux 7 (Core) 3.10.0-1160.25.1.el7.x86_64 containerd://1.4.4
trying to start up a new github integration gitpod workspace instance. But failed.
Logs from kubernetes events:
0s Normal Created pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Created container workspace 0s Normal Started pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Started container workspace 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 61m Warning Unhealthy pod/ws-cd6b74cd-e16b-40ac-a7c5-e4ea0916ad3a Readiness probe failed: Get "http://10.233.73.132:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.132:22999: connect: connection refused 60m Warning FailedToUpdateEndpoint endpoints/ws-rose-xerinae-hi47mlhr-theia Failed to update endpoint gitpod/ws-rose-xerinae-hi47mlhr-theia: Operation cannot be fulfilled on endpoints "ws-rose-xerinae-hi47mlhr-theia": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/gitpod/ws-rose-xerinae-hi47mlhr-theia, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 182be0f9-729f-4a8d-b3ee-48824504d042, UID in object meta: 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning Unhealthy pod/ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 Readiness probe failed: Get "http://10.233.73.53:22999/_supervisor/v1/status/content/wait/true": dial tcp 10.233.73.53:22999: connect: connection refused 0s Warning FailedToUpdateEndpoint endpoints/ws-kumquat-centipede-fiutrqgd-theia Failed to update endpoint gitpod/ws-kumquat-centipede-fiutrqgd-theia: Operation cannot be fulfilled on endpoints "ws-kumquat-centipede-fiutrqgd-theia": StorageError: invalid object, Code: 4, Key: /registry/services/endpoints/gitpod/ws-kumquat-centipede-fiutrqgd-theia, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e7a6b213-f593-4659-97f6-4954b8c45d5f, UID in object meta: 0s Warning FailedToUpdateEndpointSlices service/ws-kumquat-centipede-fiutrqgd-theia Error updating Endpoint Slices for Service gitpod/ws-kumquat-centipede-fiutrqgd-theia: failed to update ws-kumquat-centipede-fiutrqgd-theia-jtzcz EndpointSlice for Service gitpod/ws-kumquat-centipede-fiutrqgd-theia: endpointslices.discovery.k8s.io "ws-kumquat-centipede-fiutrqgd-theia-jtzcz" not found
pods wait status
ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Pending 0 0s ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Pending 0 0s ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 ContainerCreating 0 0s ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 ContainerCreating 0 1s ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Running 0 2s ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Error 0 10m ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Error 0 10m ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Terminating 0 10m ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 0/1 Terminating 0 10m
ws-daeomon logs
{ "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "info", "message": "received UID mapping request", "req": "{\"pid\":\"12\",\"mapping\":[{\"hostId\":33333,\"size\":1},{\"containerId\":1,\"hostId\":100000,\"size\":65534}]}", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "INFO", "time": "2021-09-13T09:55:44Z" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "containerPID": 87734, "error": "cannot find in-container PID 12 on the node", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "error", "message": "handleUIDMappingRequest: cannot find PID on host", "req": "{\"pid\":\"12\",\"mapping\":[{\"hostId\":33333,\"size\":1},{\"containerId\":1,\"hostId\":100000,\"size\":65534}]}", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "ERROR", "time": "2021-09-13T09:55:44Z" } { "level": "info", "location": "/dst/gitpod-test", "message": "Git operations complete", "severity": "INFO", "stage": "init", "time": "2021-09-13T09:55:45Z" } { "file": "github.com/opencontainers/runc/signals.go:94", "func": "main.(*signalHandler).forward", "level": "debug", "msg": "process exited", "pid": 87390, "status": 0, "time": "2021-09-13T09:55:45Z" } { "hooks": 1, "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "debug", "message": "running lifecycle hooks", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "DEBUG", "state": "ready", "time": "2021-09-13T09:55:45Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "kumquat-centipede-fiutrqgd" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "error": "open /mnt/node-cgroups/pids/kubepods-burstable-pod407881e6_4cda_47a3_be13_996c5fae064b.slice:cri-containerd:51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a/tasks: no such file or directory", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "warning", "message": "cannot read tasks file", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:55:54Z", "userId": "", "workspaceId": "kumquat-centipede-fiutrqgd" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "error": "context deadline exceeded", "instanceId": "307b1832-81e4-4511-bde1-a9f470f01679", "level": "warning", "message": "cannot wait for container", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:00Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "purple-ostrich-iaed8wg5" } { "container": "", "instanceId": "307b1832-81e4-4511-bde1-a9f470f01679", "level": "info", "message": "dispatch found new workspace container", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "INFO", "time": "2021-09-13T09:56:00Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "purple-ostrich-iaed8wg5" } { "instanceId": "307b1832-81e4-4511-bde1-a9f470f01679", "level": "error", "message": "pod disappaered from dispatch state before container was ready", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "ERROR", "time": "2021-09-13T09:56:00Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "purple-ostrich-iaed8wg5" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "error": "open /mnt/node-cgroups/pids/kubepods-burstable-pod407881e6_4cda_47a3_be13_996c5fae064b.slice:cri-containerd:51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a/tasks: no such file or directory", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "warning", "message": "cannot read tasks file", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:04Z", "userId": "", "workspaceId": "kumquat-centipede-fiutrqgd" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "error": "context deadline exceeded", "instanceId": "9874c0bf-f76c-43d7-8d1c-8765bb41b857", "level": "warning", "message": "cannot wait for container", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:14Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "purple-ostrich-iaed8wg5" } { "container": "", "instanceId": "9874c0bf-f76c-43d7-8d1c-8765bb41b857", "level": "info", "message": "dispatch found new workspace container", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "INFO", "time": "2021-09-13T09:56:14Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "purple-ostrich-iaed8wg5" } { "instanceId": "9874c0bf-f76c-43d7-8d1c-8765bb41b857", "level": "error", "message": "pod disappaered from dispatch state before container was ready", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "ERROR", "time": "2021-09-13T09:56:14Z", "userId": "bb82593a-3201-470b-9b33-58f769b6b44f", "workspaceId": "purple-ostrich-iaed8wg5" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "error": "open /mnt/node-cgroups/pids/kubepods-burstable-pod407881e6_4cda_47a3_be13_996c5fae064b.slice:cri-containerd:51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a/tasks: no such file or directory", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "warning", "message": "cannot read tasks file", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:14Z", "userId": "", "workspaceId": "kumquat-centipede-fiutrqgd" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "error": "open /mnt/node-cgroups/pids/kubepods-burstable-pod407881e6_4cda_47a3_be13_996c5fae064b.slice:cri-containerd:51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a/tasks: no such file or directory", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "warning", "message": "cannot read tasks file", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:24Z", "userId": "", "workspaceId": "kumquat-centipede-fiutrqgd" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "error": "open /mnt/node-cgroups/pids/kubepods-burstable-pod407881e6_4cda_47a3_be13_996c5fae064b.slice:cri-containerd:51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a/tasks: no such file or directory", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "warning", "message": "cannot read tasks file", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:34Z", "userId": "", "workspaceId": "kumquat-centipede-fiutrqgd" } { "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent", "containerID": "51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a", "error": "open /mnt/node-cgroups/pids/kubepods-burstable-pod407881e6_4cda_47a3_be13_996c5fae064b.slice:cri-containerd:51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a/tasks: no such file or directory", "instanceId": "b7d79f7a-b527-4205-8ea1-ef20a03a22d7", "level": "warning", "message": "cannot read tasks file", "serviceContext": { "service": "ws-daemon", "version": "" }, "severity": "WARNING", "time": "2021-09-13T09:56:44Z", "userId": "", "workspaceId": "kumquat-centipede-fiutrqgd" }
pods yaml
apiVersion: v1 kind: Pod metadata: annotations: cni.projectcalico.org/podIP: 10.233.73.53/32 cni.projectcalico.org/podIPs: 10.233.73.53/32 container.apparmor.security.beta.kubernetes.io/workspace: unconfined gitpod.io/requiredNodeServices: ws-daemon,registry-facade gitpod/admission: admit_owner_only gitpod/contentInitializer: EowBCidodHRwczovL2dpdGh1Yi5jb20vaGd5ZS9naXRwb2QtdGVzdC5naXQYAiIEbWFpbioLZ2l0cG9kLXRlc3QyTBACKkhodHRwczovL2dpdHBvZC5oaW1peC5pby9hcGkvb3RzL2dldC8xYzMyOTUwYS0zMjNiLTQ4OWYtYWRkMC1mOTVhMmZlMmZlYWQ= gitpod/customTimeout: 30m gitpod/id: b7d79f7a-b527-4205-8ea1-ef20a03a22d7 gitpod/imageSpec: CmpyZWdpc3RyeS5naXRwb2QuaGltaXguaW8vd29ya3NwYWNlLWltYWdlczpjYzNhNTBlOGVlNTk1OTA5ZWUwMGU3NjY1M2I1YmJlMzljNWVhNDc5ZWE0OGM3YjUwNWQ2YTBkYWU1YTZhNjFkEjpjb3JlLmhhcmJvci5oaW1peC5pby9naXRwb2QtaW8vc2VsZi1ob3N0ZWQvaWRlL2NvZGU6MC4xMC4w gitpod/never-ready: "true" gitpod/ownerToken: VmOIevLyFgbv4Cvfj4VSdGkONxX8kNdR gitpod/servicePrefix: kumquat-centipede-fiutrqgd gitpod/traceid: AAAAAAAAAAD591wIV5Q77Dpt789fAl2GFR+YRDqXGaEBAAAAAA== gitpod/url: https://kumquat-centipede-fiutrqgd.ws.gitpod.my-company.io prometheus.io/path: /metrics prometheus.io/port: "23000" prometheus.io/scrape: "true" seccomp.security.alpha.kubernetes.io/pod: localhost/workspace_default_0.10.0.json creationTimestamp: "2021-09-13T09:55:42Z" labels: app: gitpod component: workspace gitpod.io/networkpolicy: default gpwsman: "true" headless: "false" metaID: kumquat-centipede-fiutrqgd owner: bb82593a-3201-470b-9b33-58f769b6b44f workspaceID: b7d79f7a-b527-4205-8ea1-ef20a03a22d7 workspaceType: regular name: ws-b7d79f7a-b527-4205-8ea1-ef20a03a22d7 namespace: gitpod resourceVersion: "61509351" uid: 407881e6-4cda-47a3-be13-996c5fae064b spec: automountServiceAccountToken: false containers: - command: - /.supervisor/workspacekit - ring0 env: - name: GITPOD_REPO_ROOT value: /workspace/gitpod-test - name: GITPOD_CLI_APITOKEN value: nkM.JuEiXJELkraLK5xqGiie5R5KmRnJ - name: GITPOD_WORKSPACE_ID value: kumquat-centipede-fiutrqgd - name: GITPOD_INSTANCE_ID value: b7d79f7a-b527-4205-8ea1-ef20a03a22d7 - name: GITPOD_THEIA_PORT value: "23000" - name: THEIA_WORKSPACE_ROOT value: /workspace/gitpod-test - name: GITPOD_HOST value: https://gitpod.my-company.io - name: GITPOD_WORKSPACE_URL value: https://kumquat-centipede-fiutrqgd.ws.gitpod.my-company.io - name: THEIA_SUPERVISOR_ENDPOINT value: :22999 - name: THEIA_WEBVIEW_EXTERNAL_ENDPOINT value: webview-{{hostname}} - name: THEIA_MINI_BROWSER_HOST_PATTERN value: browser-{{hostname}} - name: GITPOD_GIT_USER_NAME value: Huaiguang Ye - name: GITPOD_GIT_USER_EMAIL value: greenar@gmail.com - name: NODE_TLS_REJECT_UNAUTHORIZED value: “0” - name: GITPOD_WORKSPACE_CONTEXT_URL value: https://github.com/hgye/gitpod-test - name: GITPOD_WORKSPACE_CONTEXT value: '{"isFile":false,"path":"","title":"hgye/gitpod-test - main","ref":"main","refType":"branch","revision":"da84f23f44fc9022bf66b3fafeac4f1008a5c269","repository":{"cloneUrl":"https://github.com/hgye/gitpod-test.git","host":"github.com","name":"gitpod-test","owner":"hgye","private":false}}' - name: GITPOD_TASKS value: '[{"init":"go get github.com/spf13/cobra/cobra\ngo build\n","command":"./mycli --help\n"}]' - name: THEIA_SUPERVISOR_TOKENS value: '[{"tokenOTS":"https://gitpod.my-company.io/api/ots/get/71c5e9fb-711a-4938-8a2b-66ccd95a4d44","token":"ots","kind":"gitpod","host":"gitpod.my-company.io","scope":["function:getWorkspace","function:getLoggedInUser","function:getPortAuthenticationToken","function:getWorkspaceOwner","function:getWorkspaceUsers","function:isWorkspaceOwner","function:controlAdmission","function:setWorkspaceTimeout","function:getWorkspaceTimeout","function:sendHeartBeat","function:getOpenPorts","function:openPort","function:closePort","function:getLayout","function:generateNewGitpodToken","function:takeSnapshot","function:storeLayout","function:stopWorkspace","function:getToken","function:getContentBlobUploadUrl","function:getContentBlobDownloadUrl","function:accessCodeSyncStorage","function:guessGitTokenScopes","function:getEnvVars","function:setEnvVar","function:deleteEnvVar","resource:workspace::kumquat-centipede-fiutrqgd::get/update","resource:workspaceInstance::b7d79f7a-b527-4205-8ea1-ef20a03a22d7::get/update/delete","resource:snapshot::ws-kumquat-centipede-fiutrqgd::create","resource:gitpodToken::*::create","resource:userStorage::*::create/get/update","resource:token::*::get","resource:contentBlob::*::create/get","resource:envVar::hgye/gitpod-test::create/get/update/delete"],"expiryDate":"2021-09-14T09:55:42.025Z","reuse":2}]' - name: GITPOD_RESOLVED_EXTENSIONS value: '{"vscode.bat@1.44.2":{"fullPluginName":"vscode.bat@1.44.2","url":"local","kind":"builtin"},"vscode.clojure@1.44.2":{"fullPluginName":"vscode.clojure@1.44.2","url":"local","kind":"builtin"},"vscode.coffeescript@1.44.2":{"fullPluginName":"vscode.coffeescript@1.44.2","url":"local","kind":"builtin"},"vscode.cpp@1.44.2":{"fullPluginName":"vscode.cpp@1.44.2","url":"local","kind":"builtin"},"vscode.csharp@1.44.2":{"fullPluginName":"vscode.csharp@1.44.2","url":"local","kind":"builtin"},"llvm-vs-code-extensions.vscode-clangd@0.1.5":{"fullPluginName":"llvm-vs-code-extensions.vscode-clangd@0.1.5","url":"local","kind":"builtin"},"vscode.css@1.51.1":{"fullPluginName":"vscode.css@1.51.1","url":"local","kind":"builtin"},"vscode.css-language-features@1.51.1":{"fullPluginName":"vscode.css-language-features@1.51.1","url":"local","kind":"builtin"},"vscode.debug-auto-launch@1.44.2":{"fullPluginName":"vscode.debug-auto-launch@1.44.2","url":"local","kind":"builtin"},"vscode.emmet@1.44.2":{"fullPluginName":"vscode.emmet@1.44.2","url":"local","kind":"builtin"},"vscode.fsharp@1.44.2":{"fullPluginName":"vscode.fsharp@1.44.2","url":"local","kind":"builtin"},"vscode.go@1.44.2":{"fullPluginName":"vscode.go@1.44.2","url":"local","kind":"builtin"},"vscode.groovy@1.44.2":{"fullPluginName":"vscode.groovy@1.44.2","url":"local","kind":"builtin"},"vscode.handlebars@1.44.2":{"fullPluginName":"vscode.handlebars@1.44.2","url":"local","kind":"builtin"},"vscode.hlsl@1.44.2":{"fullPluginName":"vscode.hlsl@1.44.2","url":"local","kind":"builtin"},"vscode.html@1.51.1":{"fullPluginName":"vscode.html@1.51.1","url":"local","kind":"builtin"},"vscode.html-language-features@1.51.1":{"fullPluginName":"vscode.html-language-features@1.51.1","url":"local","kind":"builtin"},"vscode.ini@1.44.2":{"fullPluginName":"vscode.ini@1.44.2","url":"local","kind":"builtin"},"vscode.java@1.53.2":{"fullPluginName":"vscode.java@1.53.2","url":"local","kind":"builtin"},"vscode.javascript@1.44.2":{"fullPluginName":"vscode.javascript@1.44.2","url":"local","kind":"builtin"},"vscode.json@1.44.2":{"fullPluginName":"vscode.json@1.44.2","url":"local","kind":"builtin"},"vscode.json-language-features@1.46.1":{"fullPluginName":"vscode.json-language-features@1.46.1","url":"local","kind":"builtin"},"vscode.less@1.44.2":{"fullPluginName":"vscode.less@1.44.2","url":"local","kind":"builtin"},"vscode.log@1.44.2":{"fullPluginName":"vscode.log@1.44.2","url":"local","kind":"builtin"},"vscode.lua@1.44.2":{"fullPluginName":"vscode.lua@1.44.2","url":"local","kind":"builtin"},"vscode.make@1.44.2":{"fullPluginName":"vscode.make@1.44.2","url":"local","kind":"builtin"},"vscode.markdown@1.44.2":{"fullPluginName":"vscode.markdown@1.44.2","url":"local","kind":"builtin"},"vscode.npm@1.39.1":{"fullPluginName":"vscode.npm@1.39.1","url":"local","kind":"builtin"},"vscode.objective-c@1.44.2":{"fullPluginName":"vscode.objective-c@1.44.2","url":"local","kind":"builtin"},"vscode.perl@1.44.2":{"fullPluginName":"vscode.perl@1.44.2","url":"local","kind":"builtin"},"vscode.php@1.44.2":{"fullPluginName":"vscode.php@1.44.2","url":"local","kind":"builtin"},"vscode.powershell@1.44.2":{"fullPluginName":"vscode.powershell@1.44.2","url":"local","kind":"builtin"},"vscode.pug@1.44.2":{"fullPluginName":"vscode.pug@1.44.2","url":"local","kind":"builtin"},"vscode.python@1.47.3":{"fullPluginName":"vscode.python@1.47.3","url":"local","kind":"builtin"},"vscode.r@1.44.2":{"fullPluginName":"vscode.r@1.44.2","url":"local","kind":"builtin"},"vscode.razor@1.44.2":{"fullPluginName":"vscode.razor@1.44.2","url":"local","kind":"builtin"},"vscode.ruby@1.44.2":{"fullPluginName":"vscode.ruby@1.44.2","url":"local","kind":"builtin"},"vscode.rust@1.44.2":{"fullPluginName":"vscode.rust@1.44.2","url":"local","kind":"builtin"},"vscode.scss@1.44.2":{"fullPluginName":"vscode.scss@1.44.2","url":"local","kind":"builtin"},"vscode.shaderlab@1.44.2":{"fullPluginName":"vscode.shaderlab@1.44.2","url":"local","kind":"builtin"},"vscode.shellscript@1.44.2":{"fullPluginName":"vscode.shellscript@1.44.2","url":"local","kind":"builtin"},"vscode.sql@1.44.2":{"fullPluginName":"vscode.sql@1.44.2","url":"local","kind":"builtin"},"vscode.swift@1.44.2":{"fullPluginName":"vscode.swift@1.44.2","url":"local","kind":"builtin"},"vscode.typescript@1.44.2":{"fullPluginName":"vscode.typescript@1.44.2","url":"local","kind":"builtin"},"vscode.typescript-language-features@1.44.2":{"fullPluginName":"vscode.typescript-language-features@1.44.2","url":"local","kind":"builtin"},"vscode.vb@1.44.2":{"fullPluginName":"vscode.vb@1.44.2","url":"local","kind":"builtin"},"vscode.xml@1.44.2":{"fullPluginName":"vscode.xml@1.44.2","url":"local","kind":"builtin"},"vscode.yaml@1.44.2":{"fullPluginName":"vscode.yaml@1.44.2","url":"local","kind":"builtin"},"redhat.java@0.75.0":{"fullPluginName":"redhat.java@0.75.0","url":"local","kind":"builtin"},"vscjava.vscode-java-debug@0.27.1":{"fullPluginName":"vscjava.vscode-java-debug@0.27.1","url":"local","kind":"builtin"},"vscjava.vscode-java-dependency@0.18.0":{"fullPluginName":"vscjava.vscode-java-dependency@0.18.0","url":"local","kind":"builtin"},"ms-vscode.node-debug@1.38.4":{"fullPluginName":"ms-vscode.node-debug@1.38.4","url":"local","kind":"builtin"},"ms-vscode.node-debug2@1.33.0":{"fullPluginName":"ms-vscode.node-debug2@1.33.0","url":"local","kind":"builtin"},"ms-python.python@2020.7.96456":{"fullPluginName":"ms-python.python@2020.7.96456","url":"local","kind":"builtin"},"redhat.vscode-xml@0.11.0":{"fullPluginName":"redhat.vscode-xml@0.11.0","url":"local","kind":"builtin"},"redhat.vscode-yaml@0.8.0":{"fullPluginName":"redhat.vscode-yaml@0.8.0","url":"local","kind":"builtin"},"bmewburn.vscode-intelephense-client@1.4.0":{"fullPluginName":"bmewburn.vscode-intelephense-client@1.4.0","url":"local","kind":"builtin"},"felixfbecker.php-debug@1.13.0":{"fullPluginName":"felixfbecker.php-debug@1.13.0","url":"local","kind":"builtin"},"rust-lang.rust@0.7.8":{"fullPluginName":"rust-lang.rust@0.7.8","url":"local","kind":"builtin"},"vscode.theme-abyss@1.44.2":{"fullPluginName":"vscode.theme-abyss@1.44.2","url":"local","kind":"builtin"},"vscode.theme-kimbie-dark@1.44.2":{"fullPluginName":"vscode.theme-kimbie-dark@1.44.2","url":"local","kind":"builtin"},"vscode.theme-monokai@1.44.2":{"fullPluginName":"vscode.theme-monokai@1.44.2","url":"local","kind":"builtin"},"vscode.theme-monokai-dimmed@1.44.2":{"fullPluginName":"vscode.theme-monokai-dimmed@1.44.2","url":"local","kind":"builtin"},"vscode.theme-quietlight@1.44.2":{"fullPluginName":"vscode.theme-quietlight@1.44.2","url":"local","kind":"builtin"},"vscode.theme-red@1.44.2":{"fullPluginName":"vscode.theme-red@1.44.2","url":"local","kind":"builtin"},"vscode.theme-solarized-dark@1.44.2":{"fullPluginName":"vscode.theme-solarized-dark@1.44.2","url":"local","kind":"builtin"},"vscode.theme-solarized-light@1.44.2":{"fullPluginName":"vscode.theme-solarized-light@1.44.2","url":"local","kind":"builtin"},"vscode.theme-tomorrow-night-blue@1.44.2":{"fullPluginName":"vscode.theme-tomorrow-night-blue@1.44.2","url":"local","kind":"builtin"},"vscode.vscode-theme-seti@1.44.2":{"fullPluginName":"vscode.vscode-theme-seti@1.44.2","url":"local","kind":"builtin"},"vscode.merge-conflict@1.44.2":{"fullPluginName":"vscode.merge-conflict@1.44.2","url":"local","kind":"builtin"},"ms-vscode.references-view@0.0.47":{"fullPluginName":"ms-vscode.references-view@0.0.47","url":"local","kind":"builtin"},"EditorConfig.EditorConfig@0.15.1":{"fullPluginName":"editorconfig.editorconfig@0.15.1","url":"local","kind":"builtin"},"vscode.docker@1.47.3":{"fullPluginName":"vscode.docker@1.47.3","url":"local","kind":"builtin"},"golang.go":{"fullPluginName":"golang.go@0.27.1","url":"https://open-vsx.org/api/golang/Go/0.27.1/file/golang.Go-0.27.1.vsix","kind":"workspace"}}' - name: GITPOD_EXTERNAL_EXTENSIONS value: '[]' - name: GITPOD_INTERVAL value: "30000" - name: GITPOD_MEMORY value: "2415" image: reg.gitpod.my-company.io:3000/remote/b7d79f7a-b527-4205-8ea1-ef20a03a22d7 imagePullPolicy: Always name: workspace ports: - containerPort: 23000 protocol: TCP readinessProbe: failureThreshold: 600 httpGet: path: /_supervisor/v1/status/content/wait/true port: 22999 scheme: HTTP periodSeconds: 1 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "5" memory: 12Gi requests: cpu: 1m ephemeral-storage: 5Gi memory: 2304Mi securityContext: allowPrivilegeEscalation: true capabilities: add: - AUDIT_WRITE - FSETID - KILL - NET_BIND_SERVICE - SYS_PTRACE drop: - SETPCAP - CHOWN - NET_RAW - DAC_OVERRIDE - FOWNER - SYS_CHROOT - SETFCAP - SETUID - SETGID privileged: false readOnlyRootFilesystem: false runAsGroup: 33333 runAsNonRoot: true runAsUser: 33333 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /workspace mountPropagation: HostToContainer name: vol-this-workspace - mountPath: /.workspace mountPropagation: HostToContainer name: daemon-mount dnsPolicy: ClusterFirst enableServiceLinks: false imagePullSecrets: - name: builtin-registry-auth nodeName: master.my-company.loc preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: workspace-scheduler securityContext: seccompProfile: localhostProfile: workspace_default_0.10.0.json type: Localhost serviceAccount: workspace serviceAccountName: workspace terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/disk-pressure operator: Exists - effect: NoExecute key: node.kubernetes.io/memory-pressure operator: Exists - effect: NoExecute key: node.kubernetes.io/network-unavailable operator: Exists tolerationSeconds: 30 - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - hostPath: path: /var/gitpod/workspaces/b7d79f7a-b527-4205-8ea1-ef20a03a22d7 type: DirectoryOrCreate name: vol-this-workspace - hostPath: path: /var/gitpod/workspaces/b7d79f7a-b527-4205-8ea1-ef20a03a22d7-daemon type: DirectoryOrCreate name: daemon-mount status: conditions: - lastProbeTime: null lastTransitionTime: "2021-09-13T09:55:42Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2021-09-13T09:55:42Z" message: 'containers with unready status: [workspace]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2021-09-13T09:55:42Z" message: 'containers with unready status: [workspace]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2021-09-13T09:55:42Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://51f9be1cb84b3e344e2873d1948d8eb4123e89584e928f961f6655ce20374a2a image: reg.gitpod.my-company.io:3000/remote/54aa1b6e-a92e-496d-a817-7cba64efadca:latest imageID: reg.gitpod.my-company.io:3000/remote/cd6b74cd-e16b-40ac-a7c5-e4ea0916ad3a@sha256:1c6761576c908cda6e5169176e4da66826a8e21f805447a4990f3758aa9b9d49 lastState: {} name: workspace ready: false restartCount: 0 started: true state: running: startedAt: "2021-09-13T09:55:44Z" hostIP: 172.16.10.11 phase: Running podIP: 10.233.73.53 podIPs: - ip: 10.233.73.53 qosClass: Burstable startTime: "2021-09-13T09:55:42Z"
Any idea of what happed?
Thanks
No response
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bug description
workspace failed to startup
Steps to reproduce
using helm to install gitpod self-hosted version v0.10.0, cluster node info
trying to start up a new github integration gitpod workspace instance. But failed.
Logs from kubernetes events:
pods wait status
ws-daeomon logs
pods yaml
Any idea of what happed?
Thanks
Expected behavior
No response
Example repository
No response
Anything else?
No response