Stopping Workspace Takes Ages

svenefftinge commented 3 years ago
Bug description

When a workspace stops the stopping phase takes very long (minutes). (At least in dev staging)
Steps to reproduce

In dev staging 1) Start a workspace e.g. on https://github.com/eclipse-theia/theia 2) Stop it 3) meditate (or drink a coffee)
Expected behavior

To stop in a reasonable time (< a minute would be desireable)
Example repository

https://github.com/eclipse-theia/theia
svenefftinge commented 3 years ago
I'm not sure it will stop at all (stopping for more than 10 minutes now).
svenefftinge commented 3 years ago
It seems like this only happens sometimes.
csweichel commented 3 years ago
This is still a remnant of #3608 .. fix is WIP in https://github.com/gitpod-io/gitpod/pull/3637
svenefftinge commented 3 years ago
It still happens :-(
svenefftinge commented 3 years ago
output of kubectl get pod -o yaml ...
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 10.60.9.62/32
    container.apparmor.security.beta.kubernetes.io/workspace: unconfined
    gitpod.io/requiredNodeServices: ws-daemon,registry-facade
    gitpod/admission: admit_owner_only
    gitpod/contentInitializer: EuIBCjJodHRwczovL2dpdGh1Yi5jb20vZ2l0cG9kLWlvL3N2ZWx0ZWpzLXRlbXBsYXRlLmdpdBIoaHR0cHM6Ly9naXRodWIuY29tL3N2ZWx0ZWpzL3RlbXBsYXRlLmdpdBgCIgZtYXN0ZXIqEXN2ZWx0ZWpzLXRlbXBsYXRlMmUQAiphaHR0cHM6Ly9zZS1wb3J0LW5vdC1mb3VuZC5zdGFnaW5nLmdpdHBvZC1kZXYuY29tL2FwaS9vdHMvZ2V0L2QxOTJlZWU1LTcxNjctNDVjNi04OTJkLWFlNmFjNmI3ZWIzMg==
    gitpod/customTimeout: 30m
    gitpod/id: b90659ed-34ab-491a-862a-0f99bf779081
    gitpod/imageSpec: CnRldS5nY3IuaW8vZ2l0cG9kLWNvcmUtZGV2L3JlZ2lzdHJ5L3dvcmtzcGFjZS1pbWFnZXM6M2NlM2U0OWU1NmIyMDU4ZjlhM2M5ZGEzNjFiOTkyM2E3ZjNkNTc2ZWY5ZTc4OTNhNzJhYTkzODM4ZTQwMDk5NBI9ZXUuZ2NyLmlvL2dpdHBvZC1jb3JlLWRldi9idWlsZC9pZGUvY29kZTpzZS1wb3J0LW5vdC1mb3VuZC4xMg==
    gitpod/never-ready: "true"
    gitpod/ownerToken: U[7g#Hr2T-vBvEUe;<j_G_Y6^Bj1>=.t
    gitpod/servicePrefix: plum-muskox-7ar1rg1o
    gitpod/traceid: AAAAAAAAAABO2UPo5Cl97D3f+fZ+SsHaVJQwp31si7ABAAAAAA==
    gitpod/url: https://plum-muskox-7ar1rg1o.ws-dev.se-port-not-found.staging.gitpod-dev.com
    kubernetes.io/psp: staging-se-port-not-found-ns-workspace
    prometheus.io/path: /metrics
    prometheus.io/port: "23000"
    prometheus.io/scrape: "true"
    seccomp.security.alpha.kubernetes.io/pod: localhost/workspace_default_se-port-not-found.12.json
  creationTimestamp: "2021-03-29T19:54:40Z"
  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2021-03-29T19:55:38Z"
  labels:
    app: gitpod
    component: workspace
    gitpod.io/networkpolicy: default
    gpwsman: "true"
    headless: "false"
    metaID: plum-muskox-7ar1rg1o
    owner: b9ca08d4-de70-4a33-b31e-fd8369f3763e
    workspaceID: b90659ed-34ab-491a-862a-0f99bf779081
    workspaceType: regular
  name: ws-b90659ed-34ab-491a-862a-0f99bf779081
  namespace: staging-se-port-not-found
  resourceVersion: "153941516"
  selfLink: /api/v1/namespaces/staging-se-port-not-found/pods/ws-b90659ed-34ab-491a-862a-0f99bf779081
  uid: f252f468-c518-4bef-b455-0f1ca1fb2348
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gitpod.io/workload_workspace
            operator: Exists
  automountServiceAccountToken: false
  containers:
  - command:
    - /.supervisor/workspacekit
    - ring0
    env:
    - name: GITPOD_REPO_ROOT
      value: /workspace/sveltejs-template
    - name: GITPOD_CLI_APITOKEN
      value: $hx>jT*53B67=20_=>gW>IZv0N5/<P))
    - name: GITPOD_WORKSPACE_ID
      value: plum-muskox-7ar1rg1o
    - name: GITPOD_INSTANCE_ID
      value: b90659ed-34ab-491a-862a-0f99bf779081
    - name: GITPOD_THEIA_PORT
      value: "23000"
    - name: THEIA_WORKSPACE_ROOT
      value: /workspace/sveltejs-template
    - name: GITPOD_HOST
      value: https://se-port-not-found.staging.gitpod-dev.com
    - name: GITPOD_WORKSPACE_URL
      value: https://plum-muskox-7ar1rg1o.ws-dev.se-port-not-found.staging.gitpod-dev.com
    - name: THEIA_SUPERVISOR_TOKEN
      value: 354c0b368f2b4a93b7b812564e663d23
    - name: THEIA_SUPERVISOR_ENDPOINT
      value: :22999
    - name: THEIA_WEBVIEW_EXTERNAL_ENDPOINT
      value: webview-{{hostname}}
    - name: THEIA_MINI_BROWSER_HOST_PATTERN
      value: browser-{{hostname}}
    - name: GITPOD_GIT_USER_NAME
      value: George Tsiolis
    - name: GITPOD_GIT_USER_EMAIL
      value: tsiolis.g@gmail.com
    - name: GITPOD_WORKSPACE_CONTEXT_URL
      value: https://github.com/gitpod-io/sveltejs-template
    - name: GITPOD_WORKSPACE_CONTEXT
      value: '{"isFile":false,"path":"","title":"gitpod-io/sveltejs-template - master","ref":"master","refType":"branch","revision":"ca5b93a31425f6f26a4916b498fe6d084af1ba17","repository":{"cloneUrl":"https://github.com/gitpod-io/sveltejs-template.git","host":"github.com","name":"sveltejs-template","owner":"gitpod-io","private":false,"fork":{"parent":{"cloneUrl":"https://github.com/sveltejs/template.git","host":"github.com","name":"template","owner":"sveltejs","private":false}}}}'
    - name: GITPOD_TASKS
      value: '[{"init":"npm install","command":"export CLIENT_URL=\"$(gp url 35729)/livereload.js?snipver=1&port=443\"\n{
        gp await-port 5000 && sleep 5 && gp preview $(gp url 5000) & } &> /dev/null\ngp
        open src/App.svelte\nnpm run dev\n"}]'
    - name: THEIA_SUPERVISOR_TOKENS
      value: '[{"tokenOTS":"https://se-port-not-found.staging.gitpod-dev.com/api/ots/get/c7635aa0-ee32-4c1c-af71-6ec57485d624","token":"ots","kind":"gitpod","host":"se-port-not-found.staging.gitpod-dev.com","scope":["function:getWorkspace","function:getLoggedInUser","function:getPortAuthenticationToken","function:getWorkspaceOwner","function:getWorkspaceUsers","function:isWorkspaceOwner","function:controlAdmission","function:setWorkspaceTimeout","function:getWorkspaceTimeout","function:sendHeartBeat","function:getOpenPorts","function:openPort","function:closePort","function:getLayout","function:generateNewGitpodToken","function:takeSnapshot","function:storeLayout","function:stopWorkspace","function:getToken","function:getContentBlobUploadUrl","function:getContentBlobDownloadUrl","function:accessCodeSyncStorage","function:guessGitTokenScopes","function:getEnvVars","function:setEnvVar","function:deleteEnvVar","resource:workspace::plum-muskox-7ar1rg1o::get/update","resource:workspaceInstance::b90659ed-34ab-491a-862a-0f99bf779081::get/update/delete","resource:snapshot::ws-plum-muskox-7ar1rg1o::create","resource:gitpodToken::*::create","resource:userStorage::*::create/get/update","resource:token::*::get","resource:contentBlob::*::create/get","resource:envVar::gitpod-io/sveltejs-template::create/get/update/delete"],"expiryDate":"2021-03-30T19:54:39.611Z","reuse":2}]'
    - name: GITPOD_RESOLVED_EXTENSIONS
      value: '{"vscode.bat@1.44.2":{"fullPluginName":"vscode.bat@1.44.2","url":"local","kind":"builtin"},"vscode.clojure@1.44.2":{"fullPluginName":"vscode.clojure@1.44.2","url":"local","kind":"builtin"},"vscode.coffeescript@1.44.2":{"fullPluginName":"vscode.coffeescript@1.44.2","url":"local","kind":"builtin"},"vscode.cpp@1.44.2":{"fullPluginName":"vscode.cpp@1.44.2","url":"local","kind":"builtin"},"vscode.csharp@1.44.2":{"fullPluginName":"vscode.csharp@1.44.2","url":"local","kind":"builtin"},"llvm-vs-code-extensions.vscode-clangd@0.1.5":{"fullPluginName":"llvm-vs-code-extensions.vscode-clangd@0.1.5","url":"local","kind":"builtin"},"vscode.css@1.51.1":{"fullPluginName":"vscode.css@1.51.1","url":"local","kind":"builtin"},"vscode.css-language-features@1.51.1":{"fullPluginName":"vscode.css-language-features@1.51.1","url":"local","kind":"builtin"},"vscode.debug-auto-launch@1.44.2":{"fullPluginName":"vscode.debug-auto-launch@1.44.2","url":"local","kind":"builtin"},"vscode.emmet@1.44.2":{"fullPluginName":"vscode.emmet@1.44.2","url":"local","kind":"builtin"},"vscode.fsharp@1.44.2":{"fullPluginName":"vscode.fsharp@1.44.2","url":"local","kind":"builtin"},"vscode.go@1.44.2":{"fullPluginName":"vscode.go@1.44.2","url":"local","kind":"builtin"},"vscode.groovy@1.44.2":{"fullPluginName":"vscode.groovy@1.44.2","url":"local","kind":"builtin"},"vscode.handlebars@1.44.2":{"fullPluginName":"vscode.handlebars@1.44.2","url":"local","kind":"builtin"},"vscode.hlsl@1.44.2":{"fullPluginName":"vscode.hlsl@1.44.2","url":"local","kind":"builtin"},"vscode.html@1.51.1":{"fullPluginName":"vscode.html@1.51.1","url":"local","kind":"builtin"},"vscode.html-language-features@1.51.1":{"fullPluginName":"vscode.html-language-features@1.51.1","url":"local","kind":"builtin"},"vscode.ini@1.44.2":{"fullPluginName":"vscode.ini@1.44.2","url":"local","kind":"builtin"},"vscode.java@1.53.2":{"fullPluginName":"vscode.java@1.53.2","url":"local","kind":"builtin"},"vscode.javascript@1.44.2":{"fullPluginName":"vscode.javascript@1.44.2","url":"local","kind":"builtin"},"vscode.json@1.44.2":{"fullPluginName":"vscode.json@1.44.2","url":"local","kind":"builtin"},"vscode.json-language-features@1.46.1":{"fullPluginName":"vscode.json-language-features@1.46.1","url":"local","kind":"builtin"},"vscode.less@1.44.2":{"fullPluginName":"vscode.less@1.44.2","url":"local","kind":"builtin"},"vscode.log@1.44.2":{"fullPluginName":"vscode.log@1.44.2","url":"local","kind":"builtin"},"vscode.lua@1.44.2":{"fullPluginName":"vscode.lua@1.44.2","url":"local","kind":"builtin"},"vscode.make@1.44.2":{"fullPluginName":"vscode.make@1.44.2","url":"local","kind":"builtin"},"vscode.markdown@1.44.2":{"fullPluginName":"vscode.markdown@1.44.2","url":"local","kind":"builtin"},"vscode.npm@1.39.1":{"fullPluginName":"vscode.npm@1.39.1","url":"local","kind":"builtin"},"vscode.objective-c@1.44.2":{"fullPluginName":"vscode.objective-c@1.44.2","url":"local","kind":"builtin"},"vscode.perl@1.44.2":{"fullPluginName":"vscode.perl@1.44.2","url":"local","kind":"builtin"},"vscode.php@1.44.2":{"fullPluginName":"vscode.php@1.44.2","url":"local","kind":"builtin"},"vscode.powershell@1.44.2":{"fullPluginName":"vscode.powershell@1.44.2","url":"local","kind":"builtin"},"vscode.pug@1.44.2":{"fullPluginName":"vscode.pug@1.44.2","url":"local","kind":"builtin"},"vscode.python@1.47.3":{"fullPluginName":"vscode.python@1.47.3","url":"local","kind":"builtin"},"vscode.r@1.44.2":{"fullPluginName":"vscode.r@1.44.2","url":"local","kind":"builtin"},"vscode.razor@1.44.2":{"fullPluginName":"vscode.razor@1.44.2","url":"local","kind":"builtin"},"vscode.ruby@1.44.2":{"fullPluginName":"vscode.ruby@1.44.2","url":"local","kind":"builtin"},"vscode.rust@1.44.2":{"fullPluginName":"vscode.rust@1.44.2","url":"local","kind":"builtin"},"vscode.scss@1.44.2":{"fullPluginName":"vscode.scss@1.44.2","url":"local","kind":"builtin"},"vscode.shaderlab@1.44.2":{"fullPluginName":"vscode.shaderlab@1.44.2","url":"local","kind":"builtin"},"vscode.shellscript@1.44.2":{"fullPluginName":"vscode.shellscript@1.44.2","url":"local","kind":"builtin"},"vscode.sql@1.44.2":{"fullPluginName":"vscode.sql@1.44.2","url":"local","kind":"builtin"},"vscode.swift@1.44.2":{"fullPluginName":"vscode.swift@1.44.2","url":"local","kind":"builtin"},"vscode.typescript@1.44.2":{"fullPluginName":"vscode.typescript@1.44.2","url":"local","kind":"builtin"},"vscode.typescript-language-features@1.44.2":{"fullPluginName":"vscode.typescript-language-features@1.44.2","url":"local","kind":"builtin"},"vscode.vb@1.44.2":{"fullPluginName":"vscode.vb@1.44.2","url":"local","kind":"builtin"},"vscode.xml@1.44.2":{"fullPluginName":"vscode.xml@1.44.2","url":"local","kind":"builtin"},"vscode.yaml@1.44.2":{"fullPluginName":"vscode.yaml@1.44.2","url":"local","kind":"builtin"},"redhat.java@0.75.0":{"fullPluginName":"redhat.java@0.75.0","url":"local","kind":"builtin"},"vscjava.vscode-java-debug@0.27.1":{"fullPluginName":"vscjava.vscode-java-debug@0.27.1","url":"local","kind":"builtin"},"vscjava.vscode-java-dependency@0.18.0":{"fullPluginName":"vscjava.vscode-java-dependency@0.18.0","url":"local","kind":"builtin"},"ms-vscode.node-debug@1.38.4":{"fullPluginName":"ms-vscode.node-debug@1.38.4","url":"local","kind":"builtin"},"ms-vscode.node-debug2@1.33.0":{"fullPluginName":"ms-vscode.node-debug2@1.33.0","url":"local","kind":"builtin"},"ms-python.python@2020.7.96456":{"fullPluginName":"ms-python.python@2020.7.96456","url":"local","kind":"builtin"},"golang.Go@0.14.4":{"fullPluginName":"golang.go@0.14.4","url":"local","kind":"builtin"},"redhat.vscode-xml@0.11.0":{"fullPluginName":"redhat.vscode-xml@0.11.0","url":"local","kind":"builtin"},"redhat.vscode-yaml@0.8.0":{"fullPluginName":"redhat.vscode-yaml@0.8.0","url":"local","kind":"builtin"},"bmewburn.vscode-intelephense-client@1.4.0":{"fullPluginName":"bmewburn.vscode-intelephense-client@1.4.0","url":"local","kind":"builtin"},"felixfbecker.php-debug@1.13.0":{"fullPluginName":"felixfbecker.php-debug@1.13.0","url":"local","kind":"builtin"},"rust-lang.rust@0.7.8":{"fullPluginName":"rust-lang.rust@0.7.8","url":"local","kind":"builtin"},"vscode.theme-abyss@1.44.2":{"fullPluginName":"vscode.theme-abyss@1.44.2","url":"local","kind":"builtin"},"vscode.theme-kimbie-dark@1.44.2":{"fullPluginName":"vscode.theme-kimbie-dark@1.44.2","url":"local","kind":"builtin"},"vscode.theme-monokai@1.44.2":{"fullPluginName":"vscode.theme-monokai@1.44.2","url":"local","kind":"builtin"},"vscode.theme-monokai-dimmed@1.44.2":{"fullPluginName":"vscode.theme-monokai-dimmed@1.44.2","url":"local","kind":"builtin"},"vscode.theme-quietlight@1.44.2":{"fullPluginName":"vscode.theme-quietlight@1.44.2","url":"local","kind":"builtin"},"vscode.theme-red@1.44.2":{"fullPluginName":"vscode.theme-red@1.44.2","url":"local","kind":"builtin"},"vscode.theme-solarized-dark@1.44.2":{"fullPluginName":"vscode.theme-solarized-dark@1.44.2","url":"local","kind":"builtin"},"vscode.theme-solarized-light@1.44.2":{"fullPluginName":"vscode.theme-solarized-light@1.44.2","url":"local","kind":"builtin"},"vscode.theme-tomorrow-night-blue@1.44.2":{"fullPluginName":"vscode.theme-tomorrow-night-blue@1.44.2","url":"local","kind":"builtin"},"vscode.vscode-theme-seti@1.44.2":{"fullPluginName":"vscode.vscode-theme-seti@1.44.2","url":"local","kind":"builtin"},"vscode.merge-conflict@1.44.2":{"fullPluginName":"vscode.merge-conflict@1.44.2","url":"local","kind":"builtin"},"ms-vscode.references-view@0.0.47":{"fullPluginName":"ms-vscode.references-view@0.0.47","url":"local","kind":"builtin"},"EditorConfig.EditorConfig@0.15.1":{"fullPluginName":"editorconfig.editorconfig@0.15.1","url":"local","kind":"builtin"},"vscode.docker@1.47.3":{"fullPluginName":"vscode.docker@1.47.3","url":"local","kind":"builtin"},"svelte.svelte-vscode":{"fullPluginName":"svelte.svelte-vscode@104.8.0","url":"https://open-vsx.org/api/svelte/svelte-vscode/104.8.0/file/svelte.svelte-vscode-104.8.0.vsix","kind":"workspace"}}'
    - name: GITPOD_EXTERNAL_EXTENSIONS
      value: '[]'
    - name: GITPOD_INTERVAL
      value: "30000"
    - name: GITPOD_MEMORY
      value: "2415"
    - name: THEIA_RATELIMIT_LOG
      value: "50"
    - name: SUPERVISOR_DEBUG_ENABLE
      value: "true"
    image: reg.se-port-not-found.staging.gitpod-dev.com:30824/remote/b90659ed-34ab-491a-862a-0f99bf779081
    imagePullPolicy: Always
    name: workspace
    ports:
    - containerPort: 23000
      protocol: TCP
    readinessProbe:
      failureThreshold: 600
      httpGet:
        path: /_supervisor/v1/status/content/wait/true
        port: 22999
        scheme: HTTP
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "5"
        memory: 12Gi
      requests:
        cpu: 1m
        ephemeral-storage: 5Gi
        memory: 2304Mi
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        add:
        - AUDIT_WRITE
        - FSETID
        - KILL
        - NET_BIND_SERVICE
        - SYS_PTRACE
        drop:
        - SETPCAP
        - CHOWN
        - NET_RAW
        - DAC_OVERRIDE
        - FOWNER
        - SYS_CHROOT
        - SETFCAP
        - SETUID
        - SETGID
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 33333
      runAsNonRoot: true
      runAsUser: 33333
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /workspace
      mountPropagation: HostToContainer
      name: vol-this-workspace
    - mountPath: /dev/net/tun
      name: dev-net-tun
    - mountPath: /.workspace
      mountPropagation: HostToContainer
      name: daemon-mount
  dnsConfig:
    nameservers:
    - 1.1.1.1
    - 8.8.8.8
  dnsPolicy: None
  enableServiceLinks: false
  imagePullSecrets:
  - name: gcp-sa-registry-auth
  nodeName: gke-dev-workload-1-49d27f81-hhx0
  priority: 0
  restartPolicy: Never
  schedulerName: workspace-scheduler
  securityContext:
    fsGroup: 1
    supplementalGroups:
    - 1
  serviceAccount: workspace
  serviceAccountName: workspace
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/network-unavailable
    operator: Exists
    tolerationSeconds: 30
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - hostPath:
      path: /mnt/disks/ssd0/workspaces/b90659ed-34ab-491a-862a-0f99bf779081
      type: DirectoryOrCreate
    name: vol-this-workspace
  - hostPath:
      path: /dev/net/tun
      type: File
    name: dev-net-tun
  - hostPath:
      path: /mnt/disks/ssd0/workspaces/b90659ed-34ab-491a-862a-0f99bf779081-daemon
      type: DirectoryOrCreate
    name: daemon-mount
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-03-29T19:54:40Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-03-29T19:54:44Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-03-29T19:54:44Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-03-29T19:54:40Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://fc266c93420bc2b93e4f1fc25d83bb17b60d119c2770dc941dafaf8ff646b4e0
    image: reg.main.staging.gitpod-dev.com:30723/remote/26cbd68e-3951-416d-a56f-f515a7783471:latest
    imageID: reg.se-port-not-found.staging.gitpod-dev.com:30012/remote/9286628a-a154-4560-a46d-7b043b486a88@sha256:35f282a185fe48e49e4ee878bb09bc50c889744fcf07559017a3b00b7ac50adf
    lastState: {}
    name: workspace
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-03-29T19:54:42Z"
  hostIP: 10.132.0.14
  phase: Running
  podIP: 10.60.9.62
  podIPs:
  - ip: 10.60.9.62
  qosClass: Burstable
  startTime: "2021-03-29T19:54:40Z"
csweichel commented 3 years ago
Just to make sure: you're only seeing this issue during the re-deployment of a preview environment, correct?
svenefftinge commented 3 years ago
I no longer see this at all
gitpod-io / gitpod
