loft-sh / devpod

Codespaces but open-source, client-only and unopinionated: Works with any IDE and lets you use any cloud, kubernetes or just localhost docker.
https://devpod.sh
Mozilla Public License 2.0
9.53k stars 345 forks source link

Unable to create VSCode sessions with K8s provider on Windows after upgrade #1286

Closed mkorejo closed 4 weeks ago

mkorejo commented 1 month ago

What happened?
I upgraded my DevPod version from 0.2.x to latest version, 0.5.20. Following the upgrade, I can no longer start VSCode sessions with the Kubernetes provider. VSCode Browser sessions are working. VSCode launches, the remote SSH extension kicks in, the following output is displayed, and a dialog popup is displayed saying Could not connect:

[10:46:28.582] Log Level: 1
[10:46:28.606] VS Code version: 1.93.1
[10:46:28.607] Remote-SSH version: remote-ssh@0.114.3
[10:46:28.607] win32 x64
[10:46:28.612] SSH Resolver called for "ssh-remote+murad.devpod", attempt 1
[10:46:28.613] "remote.SSH.useLocalServer": false
[10:46:28.613] "remote.SSH.useExecServer": true
[10:46:28.614] "remote.SSH.showLoginTerminal": false
[10:46:28.614] "remote.SSH.remotePlatform": {"*.devpod":"linux","10.210.98.152":"linux"}
[10:46:28.614] "remote.SSH.path": undefined
[10:46:28.614] "remote.SSH.configFile": undefined
[10:46:28.614] "remote.SSH.useFlock": true
[10:46:28.615] "remote.SSH.lockfilesInTmp": false
[10:46:28.615] "remote.SSH.localServerDownload": auto
[10:46:28.615] "remote.SSH.remoteServerListenOnSocket": false
[10:46:28.615] "remote.SSH.showLoginTerminal": false
[10:46:28.615] "remote.SSH.defaultExtensions": []
[10:46:28.616] "remote.SSH.loglevel": 1
[10:46:28.616] "remote.SSH.enableDynamicForwarding": true
[10:46:28.616] "remote.SSH.enableRemoteCommand": false
[10:46:28.616] "remote.SSH.serverPickPortsFromRange": {}
[10:46:28.616] "remote.SSH.serverInstallPath": {}
[10:46:28.616] "remote.SSH.permitPtyAllocation": false
[10:46:28.617] "remote.SSH.preferredLocalPortRange: undefined
[10:46:28.617] "remote.SSH.useCurlAndWgetConfigurationFiles: false
[10:46:28.638] SSH Resolver called for host: murad.devpod
[10:46:28.638] Setting up SSH remote "murad.devpod"
[10:46:28.646] Using commit id "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40" and quality "stable" for server
[10:46:28.652] Script variables:
 {
  "InstallExitCode.24": "AlreadyInProgress",
  "InstallExitCode.25": "ServerDownloadFailed",
  "InstallExitCode.26": "NoDownloaderAvailable",
  "InstallExitCode.27": "UnsupportedArch",
  "InstallExitCode.28": "StatusCheckFailed",
  "InstallExitCode.29": "NeedInsidersArch",
  "InstallExitCode.30": "NoDownloaderAvailableForStatusCheck",
  "InstallExitCode.31": "ServerTransferFailed",
  "InstallExitCode.32": "ServerFailedToStart",
  "InstallExitCode.33": "NeedInsidersWindows",
  "InstallExitCode.34": "CreateInstallDirFailed",
  "InstallExitCode.35": "UnsupportedPlatform",
  "InstallExitCode.36": "ServerTerminatedCVE20201416",
  "InstallExitCode.37": "UnpackFailed",
  "InstallExitCode.38": "ChangeDirFailed",
  "InstallExitCode.AlreadyInProgress": "24",
  "InstallExitCode.ServerDownloadFailed": "25",
  "InstallExitCode.NoDownloaderAvailable": "26",
  "InstallExitCode.NoDownloaderAvailableForStatusCheck": "30",
  "InstallExitCode.UnsupportedArch": "27",
  "InstallExitCode.StatusCheckFailed": "28",
  "InstallExitCode.NeedInsidersArch": "29",
  "InstallExitCode.ServerTransferFailed": "31",
  "InstallExitCode.ServerFailedToStart": "32",
  "InstallExitCode.NeedInsidersWindows": "33",
  "InstallExitCode.CreateInstallDirFailed": "34",
  "InstallExitCode.UnsupportedPlatform": "35",
  "InstallExitCode.ServerTerminatedCVE20201416": "36",
  "InstallExitCode.UnpackFailed": "37",
  "InstallExitCode.ChangeDirFailed": "38",
  "InstallUnpackCode.Success": "success",
  "InstallUnpackCode.Error": "error",
  "InstallUnpackCode.MissingFiles": "missingFiles",
  "uuid": "e7fac4deac8e",
  "startMarker": "e7fac4deac8e: running",
  "commitId": "38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40",
  "quality": "stable",
  "token": "111aa111-11a1-1111-11aa-a11aa1a1a1aa",
  "vscodeAgentFolder": "$HOME/.vscode-server",
  "allowClientDownload": "1",
  "forceClientDownload": "0",
  "cliNameInArchive": "code",
  "ignoreWgetConfigFlag": " --no-config ",
  "ignoreCurlConfigFlag": " --disable ",
  "wgetTriesSegment": "--tries=1",
  "listenArgs": "--on-host=127.0.0.1 --on-port",
  "getDownloadServerStartTrigger": "e7fac4deac8e:trigger_server_download",
  "getDownloadServerEndTrigger": "e7fac4deac8e:trigger_server_download_end",
  "getProgressDownloading": "e7fac4deac8e%%1%%",
  "getProgressInstalling": "e7fac4deac8e%%2%%"
}
[10:46:28.656] Install and start server if needed
[10:46:28.663] Checking ssh with "C:\Program Files (x86)\Common Files\Oracle\Java\javapath\ssh.exe -V"
[10:46:28.669] Got error from ssh: spawn C:\Program Files (x86)\Common Files\Oracle\Java\javapath\ssh.exe ENOENT
[10:46:28.670] Checking ssh with "C:\Windows\system32\ssh.exe -V"
[10:46:28.672] Got error from ssh: spawn C:\Windows\system32\ssh.exe ENOENT
[10:46:28.673] Checking ssh with "C:\Windows\ssh.exe -V"
[10:46:28.675] Got error from ssh: spawn C:\Windows\ssh.exe ENOENT
[10:46:28.676] Checking ssh with "C:\Windows\System32\Wbem\ssh.exe -V"
[10:46:28.680] Got error from ssh: spawn C:\Windows\System32\Wbem\ssh.exe ENOENT
[10:46:28.680] Checking ssh with "C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe -V"
[10:46:28.682] Got error from ssh: spawn C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT
[10:46:28.683] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[10:46:28.862] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5

[10:46:28.878] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 62083 "murad.devpod" sh
[10:46:28.916] Generated SSH command: 'type "D:\Users\MKorejo\AppData\Local\Temp\1\vscode-linux-multi-line-command-murad.devpod-839016287.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 62083 "murad.devpod" sh'
[10:46:28.917] Terminal shell path: C:\Windows\System32\cmd.exe
[10:46:29.857] "install" terminal received data: ""
[10:46:29.858] Got some output, clearing connection timeout
[10:46:29.957] "install" terminal received data: "CreateProcessW failed error:87[?25l
posix_spawn: Unknown error
The process tried to write to a nonexistent pipe.
[?25h"
[10:46:30.259] "install" terminal command done
[10:46:30.260] Install terminal quit with output: posix_spawn: Unknown error
[10:46:30.260] Received install output: posix_spawn: Unknown error
[10:46:30.260] Stopped parsing output early. Remaining text: posix_spawn: Unknown error
[10:46:30.261] Failed to parse remote port from server output
[10:46:30.266] Resolver error: Error: 
    at v.Create (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:501404)
    at t.handleInstallOutput (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:498726)
    at t.tryInstall (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:616460)
    at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:575788
    at async t.withShowDetailsEvent (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:579017)
    at async k (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:572574)
    at async t.resolve (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:576428)
    at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:844803
[10:46:30.282] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"","arch":"","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"0","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"0","didLocalDownload":"0","installUnpackCode":"0","outcome":"failure","reason":"UnparsableOutput","exitCodeLabel":""},"measures":{"resolveAttempts":1,"isExecServer":0,"timing.totalResolveTime":1650,"timing.preSshTime":305,"timing.scriptTime":1343}}
[10:46:30.289] ------

What did you expect to happen instead?

VSCode opens and mounts my PVC and I can start editing/working in the container.

How can we reproduce the bug? (as minimally and precisely as possible)

My devcontainer.json:

{
  "image": "public.ecr.aws/v7d9y2d2/devpod:234f16d1",
  "workspaceMount": "source=${localWorkspaceFolder},target=/root,type=bind",
  "workspaceFolder": "/root",
  "forwardPorts": [8080, 9898],
  "overrideCommand": false,
  "privileged": true
}

Local Environment:

DevPod Provider:

Anything else we need to know?
This was working prior to upgrading DevPod. I had to upgrade the provider following DevPod upgrade with devpod provider upgrade kubernetes. I've tried multiple re-installations of DevPod, Git, and VSCode, and also regenerated my SSH identity, however nothing seems to work.

I can start a remote SSH explorer session in VSCode (outside of DevPod) to another EC2 instance in my environment. VSCode Browser also works in DevPod for the same workspace which doesn't work in VSCode.

pascalbreuninger commented 1 month ago

Hey @mkorejo, thanks for reporting the issue. Is there anything else that changed since the upgrade? I wasn't able to reproduce the issue with a very basic workspace and VSCode on windows with the kubernetes provider.

Can you confirm the same behaviour also occurs if you start the workspace with https://github.com/loft-sh/devpod-example-go?

mkorejo commented 1 month ago

Sorry for the delay, @pascalbreuninger. I can confirm I get the same behavior:

[09:01:15.138] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[09:01:15.469] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5

[09:01:15.488] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63982 "devpod-example-go.devpod" sh
[09:01:15.538] Generated SSH command: 'type "D:\Users\MKorejo\AppData\Local\Temp\1\vscode-linux-multi-line-command-devpod-example-go.devpod-968841258.sh" | "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63982 "devpod-example-go.devpod" sh'
[09:01:15.542] Terminal shell path: C:\Windows\System32\cmd.exe
[09:01:18.156] "install" terminal received data: ""
[09:01:18.157] Got some output, clearing connection timeout
[09:01:18.213] "install" terminal received data: "CreateProcessW failed error:87[?25l
posix_spawn: Unknown error
The process tried to write to a nonexistent pipe.
[?25h"
[09:01:18.508] "install" terminal command done
[09:01:18.509] Install terminal quit with output: posix_spawn: Unknown error
[09:01:18.510] Received install output: posix_spawn: Unknown error
[09:01:18.510] Stopped parsing output early. Remaining text: posix_spawn: Unknown error
[09:01:18.511] Failed to parse remote port from server output
[09:01:18.514] Resolver error: Error: 
    at v.Create (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:501404)
    at t.handleInstallOutput (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:498726)
    at t.tryInstall (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:616460)
    at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:575788
    at async t.withShowDetailsEvent (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:579017)
    at async k (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:572574)
    at async t.resolve (d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:576428)
    at async d:\Users\MKorejo\.vscode\extensions\ms-vscode-remote.remote-ssh-0.114.3\out\extension.js:2:844803
[09:01:18.529] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"","arch":"","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"0","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"0","didLocalDownload":"0","installUnpackCode":"0","outcome":"failure","reason":"UnparsableOutput","exitCodeLabel":""},"measures":{"resolveAttempts":1,"isExecServer":0,"timing.totalResolveTime":3469,"timing.preSshTime":498,"timing.scriptTime":2969}}
[09:01:18.536] ------
mkorejo commented 4 weeks ago

This must've been a client-side installation issue with VSCode. It's working now.