loft-sh / devpod

Codespaces but open-source, client-only and unopinionated: Works with any IDE and lets you use any cloud, kubernetes or just localhost docker.
https://devpod.sh
Mozilla Public License 2.0
8.44k stars 311 forks source link

VSCode remote SSH to container always fails/times out in Windows #972

Open jansselt opened 3 months ago

jansselt commented 3 months ago

What happened?

devpod starts container on remote host and opens VSCode but then is never able to SSH into the container.

What did you expect to happen instead?

VSCode to open and connect to the container. This works without issue for me in Linux. In Windows it times out no matter how long I set the timeout maximum.

How can we reproduce the bug? (as minimally and precisely as possible)

For me, create any workspace and open in VSCode. I am using an SSH provider.

My devcontainer.json:

{"image":"mcr.microsoft.com/devcontainers/javascript-node"}

Local Environment:

DevPod Provider:

Anything else we need to know?

I am experiencing an issue similar to this issue I believe https://github.com/loft-sh/devpod/issues/850

The container is created just fine. I can use VSCode Browser just fine. However if I try to use VSCode, it always fails to SSH connect to the container.

[08:13:20] info Workspace resume-generator already exists
[08:13:20] debug Acquire workspace lock...
[08:13:20] debug Acquired workspace lock...
[08:13:20] info Creating devcontainer...
[08:13:20] debug Inject and run command: '/tmp/devpod/agent' helper ssh-server --stdio --debug
[08:13:20] debug Attempting to create SSH client
[08:13:20] debug execute inject script
[08:13:20] debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.4
[08:13:20] debug Run command provider command: ${SSH_PROVIDER} command
[08:13:21] debug Received line after pong: done
[08:13:22] debug SSH client created
[08:13:22] debug SSH session created
[08:13:22] info Execute SSH server command: bash -c '/tmp/devpod/agent' agent workspace up --workspace-info 'H4sIAAAAAAAA/+xTXW/iOhD9K1fznA9Ki9TmDQUu5d5uqWhY7a4iIdcegsGxI4+Tdrfiv69MoE1pV/u0b/uU2HPO+MzMmWd4NHZLFeM4s7KQGhJIkzxfEFrK8w3TRHkeCWwqI/KcG+3wyVGeC1yxWrk8f+FTnlukusSwQI2WOWM70WhDRkPw+hwkzyAFJHBKggDqk0DIV+yKQwCV5K62CAmsnasoieNCunX9ELUfRoSOIm7KuP09hENlChMOBnzwcMXOVjjoR5UufD5rGinQejGalT4x0RoCMJWTRpO/H07Gt9nybphd+1PDVO1hsSuruG1LzArUDnYBjGbp/+P5O6wwfIvWA8Zfsvlw+e/NcHLfBYTqH1Vv0VdOaO9aUQISZ2vcBXA9u8+6cFqjUlEhf5RG1HwbPUr9K+r0dphm08/T7Osym34azxY+0f7+v3GaLQ+C0/l4NL7NpsObN7J8DngFT6bZ75B3s/kbpf3+x8J2uwBKxtdSY6tHCuzMoCFuxD4hmdq2Ximkm2NlSDpjv783gJ/53q2oXHxqKW8Pn41bZH6umSyRHCsrL7HXvwh752G/n52dJ73LpHf5DQJQjNyCvOATxNl50u95xGEX/HzbXfAveNoIm9Rox6RGmxq9koXXz1/+ZMkKX2XJbVRKbg2ZldsXILDhRybFG9Yw4lZWLtSHdlTMrSGBqAtsN2sXQGvD5PmIem/RAIR51MowsZjffNxDZVYupPWRZ1EhI6T4SIybXjSILiAAqTfI3US61KJA7SRTdLTCMTraO/9DAD4hb2ffrodCou65U8jhwiclx5SCBFZM0b4jvCZnSk/0ByVnr4u791THS393+g/v9E8AAAD//wEAAP//o/eT/k0GAAA=' --debug
[08:13:22] info No root required, because neither docker nor agent daemon needs to be installed
[08:13:22] info Use /home/luke/.devpod/agent/contexts/default/workspaces/resume-generator as workspace dir
[08:13:22] debug Created logger
[08:13:22] debug Received ping from agent
[08:13:22] debug Credentials server started on port 16161...
[08:13:23] debug Incoming client connection at /
[08:13:23] debug Credentials server started...
[08:13:23] debug Wrote docker credentials helper to /home/luke/.devpod/agent/contexts/default/workspaces/resume-generator/.cache/nodoleoyznkt/docker-credential-devpod
[08:13:23] debug Workspace Folder already exists /home/luke/.devpod/agent/contexts/default/workspaces/resume-generator/content
[08:13:23] debug Workspace exists, skip downloading
[08:13:24] debug Using docker command 'docker'
[08:13:24] debug execute inject script
[08:13:24] debug Received line after pong: done
[08:13:24] debug done exec
[08:13:24] debug done inject
[08:13:24] debug done injecting
[08:13:24] debug Done InjectAgentAndExecute
[08:13:25] debug Injected into container
[08:13:25] info Setup container...
[08:13:25] debug Inject and run command: '/usr/local/bin/devpod' helper ssh-server --stdio --debug
[08:13:25] debug Attempting to create SSH client
[08:13:25] debug SSH client created
[08:13:25] debug SSH session created
[08:13:25] info Execute SSH server command: bash -c '/usr/local/bin/devpod' agent container setup --setup-info 'H4sIAAAAAAAA/7RWbW/iOBD+K8ifQ5yEBIilfuAgS9HxJl5277ZUkYkHyDaxke2w7Vb895MTWujLrfbuet/smWdmPM/YM35EPTh0Bdc05SC7gm/S7ZdU76ZU7xB5REkpMas0p1tABOWJtPM0kUKJjbYTkWMGh+TJg8Lf6IGqRKZ7XeeCATpaaF86Q/Yl0P6mBDfKEcgtsO5zHAm50LBUIBFBpQcLJYXSIk9/UJ0KrgzqoBKjIjePCO41cFUpbhBbUyhykHaFqIPKUq7R7dH6ZeTt0fov2c6LtdKpLsxZDbNwr82JL3ke9BBBElSRQ30LvJ5saJggCw1FQrMvQt6pPU3gk8hYyQLeiRxwVtwBNhzuBcN0C1zjpHKvMIMNLTKNvz/ZKnx2D5JqISsw18hCz+d4J9RPPCALRfxQ5vLbch7Po/l8MBnHZt3p9WbRfI4IKnh6T0zBr7AsOC4USOw6joPXhUIW6kWfp5Ne3OlH40W8nA0RQTut94pgvE31rliXHGdio+tqh0/JSsiAKlCYie88E5Thg2MHtn/21x8s4utoOI1m8XQyWyCC3KbbdA1g0v09msXdyfjToP9hXNoJTXaAuWAiA/Hwg98ZXq8no+hFCFPSztiEBR4v5/Zy8aneNsJJf9wpsSfUtLO4NpaFkjgzlwCrdcrJxf55e1aUi2r7LNvSHNSl3UmgON1XqP8x/+mX3uv059fR0BTZxMZrqnalbPjZyByznl/H3eEgGpc181q2Yzu2W2v4rteseY7nPmEm43HUXQwm43dw5/3JYjmPZmdy/+j149lyvBiMorg3MIqXV/MEebrP3WGnusmmBb1Ule/WD8NX4sWfU1NLrR+QhWLjX+d7fMmu6QvPj20kCq4N/mEPV+uUM0uJQiZw9aHv3NJUbkFf/fRFHy+aQQ80TbOyu5Z5Nn3WDFpJsGaNMGgy1qJNykIKwYa64IaevwnBbwaMbhLfb4WhF7Y2rcBhXhL4Xss1zHclUA0MEeQ5nl93GnXPW7gN4rSJ59stv93y2p4TfjVF1lSDiW0WhTJP5j41tqVKamAd/a6fwPa8Rui1nSD4esrnNEmGdA1VPgwO9rlp2yl7v/2+gp3aDiLo4NgNu+G9hVSFe7+JvRwU5UBRbz3oNAelab5HBC0KsGpOUBtRaW6yX3NbJGgTt1nrjxZvTQ9UppRXrNTXQtx9FzKvYOcxm4OmjGqKCLp5XKGUrRBZoe0ukXYqXp1xA1QXEhRORJ4LXi90minirdDR+mXTbaqJ+48szNw0Jtbq1ZxfIfK4Ok36an0e4CtEblZ/M8JX6PZ4LA9w/kuUB+Glow+Pc7ytSN8LZhfVv0UKYRriAaTxYlqWaU+OuZ/LF4jBv/1nHI9/AQAA//8BAAD//1/lX2S7CQAA' --container-workspace-info 'H4sIAAAAAAAA/6pWykxJVbKqVspLzE1VslIqK07OT0lVqtVRSs7J9C8oyczPKwZJg1UhyabkJ2enFuWkFoNka2sBAAAA//8BAAD///hy5ipHAAAA' --chown-workspace --debug
[08:13:25] debug Created logger
[08:13:25] debug Received ping from agent
[08:13:25] debug Start setting up container...
[08:13:25] debug Patch etc environment & profile...
[08:13:26] debug Run post create commands...
[08:13:26] debug Done setting up environment
[08:13:26] debug Setup vscode...
[08:13:26] info Install extensions 'dbaeumer.vscode-eslint' in the background
[08:13:26] debug Connection to SSH Server closed
[08:13:26] debug Done executing ssh server helper command
[08:13:26] debug Done setting up container
[08:13:26] debug done inject
[08:13:26] debug Connection to SSH Server closed
[08:13:26] debug done injecting
[08:13:26] debug Done executing ssh server helper command
[08:13:26] debug Done InjectAgentAndExecute
[08:13:26] debug Done creating devcontainer
[08:13:26] info Run 'ssh resume-generator.devpod' to ssh into the devcontainer
[08:13:26] debug No dotfiles repo specified, skipping
[08:13:26] info Starting VSCode...
[08:13:26] debug done exec
[08:13:26] debug Run vscode command code --disable-extension ms-vscode-remote.remote-containers --new-window --folder-uri=vscode-remote://ssh-remote+resume-generator.devpod//workspaces/resume-generator

Screenshot 2024-03-22 081506

Screenshot 2024-03-22 081718

This only happens for me on Windows. Linux works with no issues. I don't have a mac, so can't test that.

jansselt commented 3 months ago

Also may be related to https://github.com/loft-sh/devpod/issues/521 in which case I see the response that theres need for work on the windows SSH and potentially missing unix utilities. So that may be the cause. If so, I suspect this can be closed and I can just +1 that.

pascalbreuninger commented 3 months ago

Hi @jansselt, thanks for reporting the issue. Could you try to increase the vscode SSH timeout and see if that helps? (Second comment on this stackoverflow question: https://stackoverflow.com/questions/59978826/why-ssh-connection-timed-out-in-vscode)

jansselt commented 3 months ago

I actually stumbled across that SO while trying to figure it out myself. I did try this and it did not work for me. I let it run for 10 mins or so. And it just keeps logging "Waiting for devpod agent to come up" occasionally.

However, I will try once more later when I'm back in my network to make sure that is still the case.

pascalbreuninger commented 3 months ago

If it doesn't work out we'll take a look, thanks 👍

jansselt commented 3 months ago

Alright, let er run for about 10 mins this morning. timeout_setting

timeout_vscode_log

rodenj1 commented 3 months ago

I just wanted to add to this bug. I am running into the same issue. Windows 11 Running DevPod. Provider is ssh to a linux system. Web VScode works fine, but local VScode never connects with same error reported earlier. I did test increasing the timeout, but that did not help. Here are all the different logs

DePod image

VSCode Log image image

Doing a direct SSH to the pod also fails with the same VSCode messages image

However, if I use the DevPod CLI to connect via ssh, it works! image

pascalbreuninger commented 3 months ago

thanks @rodenj1

Analect commented 3 months ago

I can confirm the same problems per other users above ... works with vscode browser and devpod ssh ... but fails with desktop version of vscode. Set timeout from 15s to 60s. I set up provider connection as myusername@192.XXX.XXX.XXX. Just wondering, does it somehow expect a vscode user to be pre-established on the remote machine? Thanks.

[09:15:53.351] Log Level: 2
[09:15:53.373] SSH Resolver called for "ssh-remote+idpbuilder-ref-impl-idp-macmini-2204.devpod", attempt 1
[09:15:53.374] "remote.SSH.useLocalServer": false
[09:15:53.375] "remote.SSH.useExecServer": false
[09:15:53.375] "remote.SSH.showLoginTerminal": false
[09:15:53.376] "remote.SSH.remotePlatform": {"192.168.192.13":"linux","192.168.192.49":"linux","192.168.192.63":"linux","192.168.192.21":"linux","192.168.192.50":"linux","192.168.192.48":"macOS","192.168.192.27":"macOS","192.168.192.42":"linux","192.168.192.56":"linux"}
[09:15:53.376] "remote.SSH.path": undefined
[09:15:53.376] "remote.SSH.configFile": undefined
[09:15:53.376] "remote.SSH.useFlock": true
[09:15:53.377] "remote.SSH.lockfilesInTmp": false
[09:15:53.377] "remote.SSH.localServerDownload": auto
[09:15:53.377] "remote.SSH.remoteServerListenOnSocket": false
[09:15:53.377] "remote.SSH.showLoginTerminal": false
[09:15:53.378] "remote.SSH.defaultExtensions": []
[09:15:53.378] "remote.SSH.loglevel": 2
[09:15:53.378] "remote.SSH.enableDynamicForwarding": true
[09:15:53.378] "remote.SSH.enableRemoteCommand": false
[09:15:53.379] "remote.SSH.serverPickPortsFromRange": {}
[09:15:53.379] "remote.SSH.serverInstallPath": {}
[09:15:53.387] VS Code version: 1.87.2
[09:15:53.387] Remote-SSH version: remote-ssh@0.109.0
[09:15:53.387] win32 x64
[09:15:53.485] SSH Resolver called for host: idpbuilder-ref-impl-idp-macmini-2204.devpod
[09:15:53.485] Setting up SSH remote "idpbuilder-ref-impl-idp-macmini-2204.devpod"
[09:15:53.511] Using commit id "863d2581ecda6849923a2118d93a088b0745d9d6" and quality "stable" for server
[09:15:53.519] Install and start server if needed
[09:15:56.238] Checking ssh with "C:\Program Files (x86)\Google\Chrome\Application\ssh.exe -V"
[09:15:56.241] Got error from ssh: spawn C:\Program Files (x86)\Google\Chrome\Application\ssh.exe ENOENT
[09:15:56.242] Checking ssh with "C:\Program Files (x86)\VMware\VMware Workstation\bin\ssh.exe -V"
[09:15:56.247] Got error from ssh: spawn C:\Program Files (x86)\VMware\VMware Workstation\bin\ssh.exe ENOENT
[09:15:56.247] Checking ssh with "C:\WINDOWS\system32\ssh.exe -V"
[09:15:56.254] Got error from ssh: spawn C:\WINDOWS\system32\ssh.exe ENOENT
[09:15:56.254] Checking ssh with "C:\WINDOWS\ssh.exe -V"
[09:15:56.257] Got error from ssh: spawn C:\WINDOWS\ssh.exe ENOENT
[09:15:56.258] Checking ssh with "C:\WINDOWS\System32\Wbem\ssh.exe -V"
[09:15:56.260] Got error from ssh: spawn C:\WINDOWS\System32\Wbem\ssh.exe ENOENT
[09:15:56.260] Checking ssh with "C:\WINDOWS\System32\WindowsPowerShell\v1.0\ssh.exe -V"
[09:15:56.263] Got error from ssh: spawn C:\WINDOWS\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT
[09:15:56.263] Checking ssh with "C:\WINDOWS\System32\OpenSSH\ssh.exe -V"
[09:15:56.340] > OpenSSH_for_Windows_8.6p1, 
[09:15:56.341] > LibreSSL 3.4.3

[09:15:56.355] Running script with connection command: "C:\WINDOWS\System32\OpenSSH\ssh.exe" -T -D 11088 "idpbuilder-ref-impl-idp-macmini-2204.devpod" bash
[09:15:56.357] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[09:16:17.081] > 09:16:17infoWaiting for devpod agent to come up...
> ]0;C:\WINDOWS\System32\cmd.exe
[09:16:17.081] Got some output, clearing connection timeout
[09:16:40.110] > 09:16:40 info Waiting for devpod agent to come up...
[09:17:03.137] > 09:17:03 info Waiting for devpod agent to come up...
[09:17:26.147] > 09:17:26 info Waiting for devpod agent to come up...
[09:17:49.163] > 09:17:49 info Waiting for devpod agent to come up...
eduyubero commented 2 months ago

+1

I am facing the exact same problem my peers have.

The following is my setup:

Results are the same described above.

jzazo commented 2 months ago

Same experience as me. I am connecting via tunnel to my remote machine and I am forwarding ssh ports manually (2222). Web vscode works though.

fzxu commented 2 months ago

I have the same problem. vscode shows 'Opening Remote...' forever.

...
[23:10:38] debug Created logger
[23:10:38] debug Received ping from agent
[23:10:38] debug Start setting up container...
[23:10:39] debug Credentials server started on port 15235...
[23:10:40] debug Incoming client connection at /
[23:10:40] debug Credentials server started...
[23:10:40] debug Patch etc environment & profile...
[23:10:40] debug Run post create commands...
[23:10:41] debug Done setting up container
[23:10:41] debug Connection to SSH Server closed
[23:10:41] debug Done executing ssh server helper command
[23:10:41] debug Done creating devcontainer
[23:10:41] info Run 'ssh abc-web.devpod' to ssh into the devcontainer
[23:10:41] debug No dotfiles repo specified, skipping
[23:10:41] info Starting VSCode...
[23:10:42] debug Run vscode command code --disable-extension ms-vscode-remote.remote-containers --new-window --folder-uri=vscode-remote://ssh-remote+abc-web.devpod//workspaces/abc-web

Windows 11 Pro 23H2 64-bit operating system, x64-based processor

Please looking into this or if there is any workaround. Really appreciated.

89luca89 commented 2 months ago

Hi all!

Can you please attach the following tests:

Thanks a lot!

fzxu commented 2 months ago

devpod ssh --debug name-of-project.devpod is working fine from my knowledge, it can ssh and keep output 'Start refresh' token or something.

try from vscode, there is no error, it just hang at Install and start server if needed

[00:20:16.811] Log Level: 2
[00:20:16.819] SSH Resolver called for "ssh-remote+abc.devpod", attempt 1
[00:20:16.819] "remote.SSH.useLocalServer": false
[00:20:16.819] "remote.SSH.useExecServer": true
[00:20:16.819] "remote.SSH.showLoginTerminal": false
[00:20:16.819] "remote.SSH.remotePlatform": {}
[00:20:16.819] "remote.SSH.path": undefined
[00:20:16.820] "remote.SSH.configFile": undefined
[00:20:16.820] "remote.SSH.useFlock": true
[00:20:16.820] "remote.SSH.lockfilesInTmp": false
[00:20:16.820] "remote.SSH.localServerDownload": auto
[00:20:16.820] "remote.SSH.remoteServerListenOnSocket": false
[00:20:16.820] "remote.SSH.showLoginTerminal": false
[00:20:16.822] "remote.SSH.defaultExtensions": []
[00:20:16.822] "remote.SSH.loglevel": 2
[00:20:16.822] "remote.SSH.enableDynamicForwarding": true
[00:20:16.822] "remote.SSH.enableRemoteCommand": false
[00:20:16.823] "remote.SSH.serverPickPortsFromRange": {}
[00:20:16.823] "remote.SSH.serverInstallPath": {}
[00:20:16.830] VS Code version: 1.89.0
[00:20:16.830] Remote-SSH version: remote-ssh@0.110.1
[00:20:16.830] win32 x64
[00:20:16.834] SSH Resolver called for host: abc.devpod
[00:20:16.834] Setting up SSH remote "abc.devpod"
[00:20:16.838] Using commit id "b58957e67ee1e712cebf466b995adf4c5307b2bd" and quality "stable" for server
[00:20:16.840] Install and start server if needed

I have --debug added but seems no more logs. I also increased the remote ssh timeout to 60s from 15s.

jzazo commented 2 months ago

Yeah, I confirm I can ssh with devpod ssh name-of-project.devpod as well.

fzxu commented 2 months ago

update from my side. It was actually a false alarm from my side. For windows there is additional selection popup (I didn't noticed). After choosing 'Linux' it works for me. Thanks!

omniproc commented 8 hours ago

Can confirm I'm running into the same issue on Windows 10 22H2 Build 19045.4529 with OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 using the DevPod v.0.5.16 with SSH provider v0.0.12 instead of latest because of this bug. devpod ssh name-of-project works fine and vscode web works fine too. The issue seems only to be present when using vscode (native). Can provide logs but they are exactly the same as already posted a few times here. It hangs at Waiting for devpod agent to come up.