Closed raine-works closed 1 month ago
@raine-works are you experiencing the same problem with the latest v0.5.6 release?
I updated to v0.5.6 and I am still seeing the same error.
PS C:\Users\Raine> ssh homelab.devpod
13:40:39 info Waiting for devpod agent to come up...
13:41:02 info Waiting for devpod agent to come up...
I tried deleting that workspace and creating a new one after updating. I am seeing the same results. The devpod cli ssh command works but the standard ssh command does not.
Hi @raine-works can you please to a test for me?
Can you open the ssh config file (It should be $HOME\.ssh\config
also on windows) and add a --debug
to the ProxyCommand of devpod?
Like:
...
...
ProxyCommand devpod ssh --debug --stdio --context default --user vscode distrobox
User vscode
So we can get some more useful logs from your command?
Here is my SSH config.
# DevPod Start hukt.devpod
Host hukt.devpod
ForwardAgent yes
LogLevel error
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --debug --stdio --context default --user raine hukt
User raine
# DevPod End hukt.devpod
Host virgo.lab.raineworks.com
HostName virgo.lab.raineworks.com
User raine
Here are the logs I am seeing after adding the --debug flag.
PS C:\Users\Raine> ssh hukt.devpod
12:21:05 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
12:21:05 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
12:21:05 debug Error adding key C:\Users\Raine\.ssh\id_rsa to agent: Error connecting to agent: No such file or directory
exit status 2
12:21:05 debug Acquire workspace lock...
12:21:05 debug Acquired workspace lock...
12:21:05 debug execute inject script
12:21:05 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
12:21:05 debug Run command provider command: ${SSH_PROVIDER} command
12:21:25 debug done inject
12:21:25 debug done injecting
12:21:25 info Waiting for devpod agent to come up...
The "inject script" portion repeats endlessly.
Thanks @raine-works can you update the ssh provider to the latest version? we implemented some fixes for how it behaves and those could potentially help for this issue
I assume this can be done by deleting the provider and recreating it? I did that and Its still not working. I am seeing the same error message.
Can you post it?
PS C:\Users\Raine> ssh hukt.devpod
11:53:18 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
11:53:18 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
11:53:18 debug Error adding key C:\Users\Raine\.ssh\id_rsa to agent: Error connecting to agent: No such file or directory
exit status 2
11:53:18 debug Acquire workspace lock...
11:53:18 debug Acquired workspace lock...
11:53:18 debug execute inject script
11:53:18 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
11:53:18 debug Run command provider command: ${SSH_PROVIDER} command
11:53:38 debug done inject
11:53:38 debug done injecting
11:53:38 info Waiting for devpod agent to come up...
11:53:38 debug Inject Error: context deadline exceeded
11:53:38 debug done exec
11:53:41 debug execute inject script
11:53:41 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
11:53:41 debug Run command provider command: ${SSH_PROVIDER} command
Host hukt.devpod
ForwardAgent yes
LogLevel error
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --debug --stdio --context default --user raine hukt
User raine
# DevPod End hukt.devpod
Host virgo.lab.raineworks.com
HostName virgo.lab.raineworks.com
User raine
@raine-works, what OpenSSH version are you using? ssh -V
The default OpenSSH version from Windows is VERY old (8.1), so downloading the latest release from PowerShell/Win32-OpenSSH is probably necessary. Alternatively, if you don't make use of the SSH agent, you can disable the Windows service to fix it as well.
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3
Would you be open to upgrading by following this guide from Microsoft? https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH
It's most likely that the source (8.6) is incompatible with the target (likely 9.x).
I updated to OpenSSH version 9.5.
PS C:\Users\Raine> ssh -V
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
I am still seeing a similar error.
PS C:\Users\Raine> ssh hukt.devpod
10:49:24 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
10:49:24 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
10:49:24 debug Acquire workspace lock...
10:49:24 debug Acquired workspace lock...
10:49:24 debug execute inject script
10:49:24 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:49:24 debug Run command provider command: ${SSH_PROVIDER} command
10:49:44 debug done inject
10:49:44 debug done injecting
10:49:44 info Waiting for devpod agent to come up...
10:49:44 debug Inject Error: context deadline exceeded
10:49:44 debug done exec
10:49:47 debug execute inject script
10:49:47 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:49:47 debug Run command provider command: ${SSH_PROVIDER} command
So that "error" might happen a couple of times while the agent process is getting started. Then it normally goes away once the agent is ready to accept commands. Does it just continue to spin like that indefinitely?
The longest I've let it run is for about 5 minutes. This is the log output after 3 minutes.
PS C:\Users\Raine> ssh hukt.devpod
10:57:09 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
10:57:09 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
10:57:09 debug Acquire workspace lock...
10:57:09 debug Acquired workspace lock...
10:57:09 debug execute inject script
10:57:09 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:09 debug Run command provider command: ${SSH_PROVIDER} command
10:57:29 debug done inject
10:57:29 debug done injecting
10:57:29 info Waiting for devpod agent to come up...
10:57:29 debug Inject Error: context deadline exceeded
10:57:29 debug done exec
10:57:32 debug execute inject script
10:57:32 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:32 debug Run command provider command: ${SSH_PROVIDER} command
10:57:52 debug done inject
10:57:52 debug done injecting
10:57:52 info Waiting for devpod agent to come up...
10:57:52 debug Inject Error: context deadline exceeded
10:57:52 debug done exec
10:57:55 debug execute inject script
10:57:55 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:55 debug Run command provider command: ${SSH_PROVIDER} command
10:58:15 debug done inject
10:58:15 debug done injecting
10:58:15 info Waiting for devpod agent to come up...
10:58:15 debug Inject Error: context deadline exceeded
10:58:15 debug done exec
10:58:18 debug execute inject script
10:58:18 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:58:18 debug Run command provider command: ${SSH_PROVIDER} command
10:58:38 debug done inject
10:58:38 debug done injecting
10:58:38 info Waiting for devpod agent to come up...
10:58:38 debug Inject Error: context deadline exceeded
10:58:38 debug done exec
10:58:41 debug execute inject script
10:58:41 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:58:41 debug Run command provider command: ${SSH_PROVIDER} command
10:59:01 debug done inject
10:59:01 debug done injecting
10:59:01 info Waiting for devpod agent to come up...
10:59:01 debug Inject Error: context deadline exceeded
10:59:01 debug done exec
10:59:04 debug execute inject script
10:59:04 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:04 debug Run command provider command: ${SSH_PROVIDER} command
10:59:24 debug done inject
10:59:24 debug done injecting
10:59:24 info Waiting for devpod agent to come up...
10:59:24 debug Inject Error: context deadline exceeded
10:59:24 debug done exec
10:59:27 debug execute inject script
10:59:27 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:27 debug Run command provider command: ${SSH_PROVIDER} command
10:59:47 debug done inject
10:59:47 debug done injecting
10:59:47 info Waiting for devpod agent to come up...
10:59:47 debug Inject Error: context deadline exceeded
10:59:47 debug done exec
10:59:50 debug execute inject script
10:59:50 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:50 debug Run command provider command: ${SSH_PROVIDER} command
11:00:10 debug done inject
11:00:10 debug done injecting
11:00:10 info Waiting for devpod agent to come up...
11:00:10 debug Inject Error: context deadline exceeded
11:00:10 debug done exec
I think there is a typo in 'https://github.com/loft-sh/devpod/releases/download/v0.5.6', the URL is 404.
The default agent URL under general in the settings is https://github.com/loft-sh/devpod/releases/latest/download/
which redirects to https://github.com/loft-sh/devpod/releases/download/v0.5.6
. What should the override agent URL be?
So that download URL is just the first fragment. The file name gets appended to it based upon the target systems OS and architecture.
It was a little misleading when I was debugging that area as well, but that part is just fine from what I've seen.
I removed the agent from the remote machine and tried re-creating the workspace. The agent was successfully re-installed on the remote machine, but still hangs when trying to open a connection.
I find it strange that the devpod cli ssh command can connect to the remote devcontainer but open-ssh in powershell can't.
PowerShell 7.4.2
Raine ~ ♥ 16:04 ssh node-test.devpod
16:04:29 info Waiting for devpod agent to come up...
16:04:52 info Waiting for devpod agent to come up...
16:05:15 info Waiting for devpod agent to come up...
16:05:38 info Waiting for devpod agent to come up...
16:06:01 info Waiting for devpod agent to come up...
16:06:24 info Waiting for devpod agent to come up...
16:06:47 info Waiting for devpod agent to come up...
16:07:10 info Waiting for devpod agent to come up...
16:07:33 info Waiting for devpod agent to come up...
16:07:56 info Waiting for devpod agent to come up...
16:08:19 info Waiting for devpod agent to come up...
16:08:42 info Waiting for devpod agent to come up...
16:09:05 info Waiting for devpod agent to come up...
16:09:28 fatal connect to server: timeout waiting for instance connection: context deadline exceeded
Raine ~ ♥ 16:09 devpod node-test
16:10:35 error Try using the --debug flag to see a more verbose output
16:10:35 fatal unknown command "node-test" for "devpod"
Raine ~ ♥ 16:10 devpod ssh node-test
node ➜ ~ $ ls
node ➜ ~ $ ls -a
. .bash_logout .cache .docker .npm .profile .zprofile
.. .bashrc .config .gitconfig .oh-my-zsh .vscode-server .zshrc
node ➜ ~ $ cd /workspaces/
node ➜ /workspaces $ ls
node-test
node ➜ /workspaces $ cd node-test/
node ➜ /workspaces/node-test (main) $ ls
CODE_OF_CONDUCT.md LICENSE README.md SECURITY.md node_modules package.json server.js yarn.lock
node ➜ /workspaces/node-test (main) $ npm run start
> docker_web_app@1.0.0 start
> node server.js
Running on http://0.0.0.0:3000
@shanman190
I copied the SSH config from the Windows file system and pasted it in the SSH config in WSL. I adjusted the proxy command to use /mnt/c.
# DevPod Start node-test.devpod
Host node-test.devpod
ForwardAgent yes
LogLevel error
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
ProxyCommand "/mnt/c/Program Files/DevPod/devpod-cli.exe" ssh --stdio --context default --user node node-test
User node
# DevPod End node-test.devpod
In WSL it worked immediately.
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022
My issue must be related to the open-ssh version like you suggested before. Although updating to 9.x didn't solve the problem.
@raine-works try using the new version of the ssh provider, I had the same problem as you. The new version fixed the problem for me.
(Even though it introduced another bug :/)
Hallelujah! It works!!!!
What happened?
Attempting to connect to provider remote host via SSH times out with error: "Waiting for Devpod agent to come up..." Cannot connect via SSH with VSCode or the terminal. The Devpod CLI SSH command works as intended.
What did you expect to happen instead?
Devcontainer to be accessible via SSH.
How can we reproduce the bug? (as minimally and precisely as possible)
My
devcontainer.json
:My
Dockerfile
:My SSH
config
:Local Environment:
devpod 0.5.5
DevPod Provider:
Anything else we need to know?
I am able to connect to the remote host via SSH directly.
Docker engine on remote host has the following custom remote access config: