0.262.3 Clone Repository in Container Volume fails ssh auth for private repositories

[gagemillerlob]

• VSCode Version: 1.73
• Local OS Version: Windows 11 21H2
• Remote OS Version: Alpine 3.15 (ANY)
• Remote Extension/Connection Type: Containers v0.262.3
• Logs: (Included)

Steps to Reproduce: NOTE: Ensure that there is no existing volume for the repo

1. Create a private repository
2. Setup SSH keys for user
3. Add ssh key to ssh agent and setup git config
4. Run Clone Repository in Container Volume and use the SSH url for the private repo

Recently I have been running into issues cloning private repositories using SSH keys when using the latest release of Dev Containers. I was been able to produce this result in both gitlab and github using different ssh keys and cleared my ssh identities from my ssh agent.

Github

Gitlab

When I switch to version 0.255.4, the clone works no problem. Switching back to the latest release or Pre-Release versions causes the error to reappear.

Logs:

Github with 0.262.3

[111 ms] Dev Containers 0.262.3 in VS Code 1.73.0 (8fa188b2b301d36553cbc9ce1b0a146ccb93351f).
[110 ms] Start: Resolving Remote
[135 ms] Start: Check Docker is running
[135 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[351 ms] Server API version: 1.41
[352 ms] Start: Run: docker volume ls -q
[538 ms] Start: Run: docker volume create --label vsch.local.repository=git@github.com:gagemillerlob/private-test.git --label vsch.local.repository.unique=true private-test-03553914801430e9b57cc59a2ef8a1c1
[725 ms] Start: Run: docker build -f C:\Users\GAGEMI~1\AppData\Local\Temp\vsch\bootstrap-image\0.262.3\bootstrap.Dockerfile -t vsc-volume-bootstrap C:\Users\GAGEMI~1\AppData\Local\Temp\vsch\bootstrap-image\0.262.3

[+] Building 0.4s (9/9) FINISHED
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 42B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for mcr.microsoft.com/devcontainers/base:0-a  0.2s
 => [internal] load build context                                          0.0s
 => => transferring context: 5.90kB                                        0.0s
 => [1/4] FROM mcr.microsoft.com/devcontainers/base:0-alpine-3.16@sha256:  0.0s
 => CACHED [2/4] RUN apk add --no-cache  nodejs  python3  npm  make  g++   0.0s
 => CACHED [3/4] RUN cd && npm i node-pty                                  0.0s
 => CACHED [4/4] COPY .vscode-remote-containers /root/.vscode-remote-cont  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:48aa9733d35e2f220937c35bd96ac33273a48611f8b87  0.0s
 => => naming to docker.io/library/vsc-volume-bootstrap                    0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l
earn how to fix them
[2156 ms] Cloning Github repository: gagemillerlob/private-test into /workspaces/private-test

[2157 ms] Start: Run: docker run -d --mount type=volume,src=private-test-03553914801430e9b57cc59a2ef8a1c1,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[2909 ms] Start: Run in container: /bin/sh
[2920 ms] Start: Launching Dev Containers helper.
[2921 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-2bcb76edc5af986f222295514a22000db53c2eb9.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[2921 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[2927 ms] findLocalWindowsExecutable: Exectuable 'gpgconf' not found on PATH 'C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Program Files\Docker\Docker\resources\bin;C:\Users\GageMiller\AppData\Local\Microsoft\WindowsApps;C:\Users\GageMiller\AppData\Local\JetBrains\Toolbox\scripts;C:\Users\GageMiller\AppData\Local\Programs\Microsoft VS Code\bin'.
[2929 ms] Start: Run in container: /bin/sh
[2942 ms] Start: Run in container: echo ~
[3287 ms] /root
[3287 ms] 
[3288 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-2bcb76edc5af986f222295514a22000db53c2eb9.js' >/tmp/vscode-remote-containers-2bcb76edc5af986f222295514a22000db53c2eb9.js
[3292 ms] 
[3293 ms] 
[3293 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-2bcb76edc5af986f222295514a22000db53c2eb9.js' >/tmp/vscode-remote-containers-server-2bcb76edc5af986f222295514a22000db53c2eb9.js_1667777180692
[3297 ms] 
[3297 ms] 
[3299 ms] Start: Run in container: # Test for /root/.gitconfig and git
[3302 ms] 
[3302 ms] 
[3302 ms] Start: Run in container: # Copy C:\Users\GageMiller\.gitconfig to /root/.gitconfig
[3305 ms] 
[3306 ms] 
[3306 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global --replace-all credential.helper '!f() { node /tmp/vscode-remote-containers-2bcb76edc5af986f222295514a22000db53c2eb9.js $*; }; f' || true
[3309 ms] 
[3309 ms] 
[3309 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[3312 ms] 
[3312 ms] 
[3312 ms] Start: Run in container: # Copy C:\Users\GageMiller\.ssh\known_hosts to /root/.ssh/known_hosts
[3315 ms] 
[3315 ms] 
[3378 ms] Start: Run in container: git clone --depth 1 git@github.com:gagemillerlob/private-test.git .
Cloning into '.'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[4133 ms] Start: Run: docker rm -f ae2eba3ad0243401802ca000f47a409d6e4bc944b9a7773642c20d4122a66abf
[4468 ms] Container server terminated (code: 137, signal: null).

Github with 0.255.4

[83 ms] Dev Containers 0.255.4 in VS Code 1.73.0 (8fa188b2b301d36553cbc9ce1b0a146ccb93351f).
[82 ms] Start: Resolving Remote
[101 ms] Start: Check Docker is running
[101 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[364 ms] Server API version: 1.41
[365 ms] Start: Run: docker volume ls -q
[581 ms] Start: Run: docker build -f C:\Users\GAGEMI~1\AppData\Local\Temp\vsch\bootstrap-image\0.255.4\bootstrap.Dockerfile -t vsc-volume-bootstrap C:\Users\GAGEMI~1\AppData\Local\Temp\vsch\bootstrap-image\0.255.4

[+] Building 0.4s (9/9) FINISHED
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 42B                                        0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for mcr.microsoft.com/vscode/devcontainers/b  0.2s
 => [1/4] FROM mcr.microsoft.com/vscode/devcontainers/base:0-alpine-3.14@  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 5.90kB                                        0.0s
 => CACHED [2/4] RUN apk add --no-cache  nodejs  npm  make  g++  docker-c  0.0s
 => CACHED [3/4] RUN cd && npm i node-pty                                  0.0s
 => CACHED [4/4] COPY .vscode-remote-containers /root/.vscode-remote-cont  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:18039086c02022767b4932cde1105950a6f8c693b9e96  0.0s
 => => naming to docker.io/library/vsc-volume-bootstrap                    0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and l
earn how to fix them
[2032 ms] Cloning Github repository: gagemillerlob/private-test into /workspaces/private-test

[2033 ms] Start: Run: docker run -d --mount type=volume,src=private-test-03553914801430e9b57cc59a2ef8a1c1,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[2830 ms] Start: Run in container: /bin/sh
[2841 ms] Start: Launching Dev Containers helper.
[2841 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-caec096d5c6c1ae1f41562f08241fefdc7c6c3fc.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[2841 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[2850 ms] findLocalWindowsExecutable: Exectuable 'gpgconf' not found on PATH 'C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Git\cmd;C:\Program Files\dotnet\;C:\Program Files\Docker\Docker\resources\bin;C:\Users\GageMiller\AppData\Local\Microsoft\WindowsApps;C:\Users\GageMiller\AppData\Local\JetBrains\Toolbox\scripts;C:\Users\GageMiller\AppData\Local\Programs\Microsoft VS Code\bin'.
[2853 ms] Start: Run in container: /bin/sh
[2864 ms] Start: Run in container: echo ~
[3210 ms] /root
[3210 ms] 
[3211 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-caec096d5c6c1ae1f41562f08241fefdc7c6c3fc.js' >/tmp/vscode-remote-containers-caec096d5c6c1ae1f41562f08241fefdc7c6c3fc.js
[3219 ms] 
[3219 ms] 
[3219 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-caec096d5c6c1ae1f41562f08241fefdc7c6c3fc.js' >/tmp/vscode-remote-containers-server-caec096d5c6c1ae1f41562f08241fefdc7c6c3fc.js_1667776992726
[3224 ms] 
[3224 ms] 
[3226 ms] Start: Run in container: # Test for /root/.gitconfig and git
[3228 ms] 
[3228 ms] 
[3228 ms] Start: Run in container: # Copy C:\Users\GageMiller\.gitconfig to /root/.gitconfig
[3231 ms] 
[3231 ms] 
[3231 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global --replace-all credential.helper '!f() { node /tmp/vscode-remote-containers-caec096d5c6c1ae1f41562f08241fefdc7c6c3fc.js $*; }; f' || true
[3238 ms] 
[3238 ms] 
[3239 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[3241 ms] 
[3241 ms] 
[3241 ms] Start: Run in container: # Copy C:\Users\GageMiller\.ssh\known_hosts to /root/.ssh/known_hosts
[3244 ms] 
[3245 ms] 
[3405 ms] Start: Run in container: git clone --depth 1 git@github.com:gagemillerlob/private-test.git .
Cloning into '.'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
[4549 ms] Start: Run in container: cat /workspaces/private-test/.devcontainer/devcontainer.json 2>/dev/null
[4552 ms] Start: Run in container: cat /workspaces/private-test/.devcontainer.json 2>/dev/null

[cadsuara]

I was facing exactly the same problem when trying to "Clone Repository in Named Container Volume..."

Version: 1.73.0 (user setup) Commit: 8fa188b2b301d36553cbc9ce1b0a146ccb93351f Date: 2022-11-01T15:34:06.111Z Electron: 19.0.17 Chromium: 102.0.5005.167 Node.js: 16.14.2 V8: 10.2.154.15-electron.0 OS: Windows_NT x64 10.0.19044 Sandboxed: No

Switching back to Dev Containers version 0.255.4 seems like a viable workaround for the time being

[gagemillerlob]

I asked some colleagues to reproduce the issue on Mac and they did not see the error. Seems like this might only affect Windows.

[skurpi]

Just had the same issue on Windows Version: 10.0.19044 Build 19044 VSCode: 1.72.2 Going back from 0.262.3 to 0.255.4 solved it for me. Thanks for that.

[chrmarti]

Could you append your ~/.gitconfig and the output from git config --list?

[cadsuara]

Could you append your ~/.gitconfig and the output from git config --list?

[user]
name = Carlos Adsuara Anglés
email = cadsuara@maxlinear.com

PS C:\Users\cadsuara> git config --list
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=schannel
core.autocrlf=true
core.fscache=true
core.symlinks=true
pull.rebase=false
init.defaultbranch=master
user.name=Carlos Adsuara Anglés
user.email=cadsuara@maxlinear.com

[gagemillerlob]

Could you append your ~/.gitconfig and the output from git config --list?

[user]
    email = gage.miller@liveoak.bank
    name = gagemillerlob

diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
pull.rebase=false
credential.helper=manager-core
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.email=gage.miller@liveoak.bank
:...skipping...
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
pull.rebase=false
credential.helper=manager-core
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.email=gage.miller@liveoak.bank
user.name=gagemillerlob

[skurpi]

$> cat ~/.gitconfig
[user]
        email = dario.vodopivec@gmail.com
        name = skurpi
$> git config --list
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
pull.rebase=false
credential.helper=manager-core
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.email=dario.vodopivec@gmail.com
user.name=skurpi

[akutruff]

UGH. I've been tearing my hair out trying to get my dev environments back. What other info do you need?

[gagemillerlob]

UGH. I've been tearing my hair out trying to get my dev environments back. What other info do you need?

I'd suggest rolling back for now. Version 0.255 is functional enough until a fix is pushed.

[makeable]

I guess they dont use windows at microsoft :D

Happy to fix this, but cant find the code anywhere.... Can someone point me in the right direction - this repo structure is very confusing, seemingly just used for issues.

[chrmarti]

Not sure what has changed here. Could you run the following commands locally and post the output here?

ssh -T git@github.com
git clone git@github.com:gagemillerlob/private-test.git
echo $env:SSH_AUTH_SOCK

[jsedanoj]

I am the reporter of the linked issue #7539 (which is a duplicate of this one).

I cannot do the test requested to gagemillerlob because git clone git@github.com:gagemillerlob/private-test.git is not found (or not public), but I can do a similar test if you want.

Also, notice that in the linked issue #7539 I say that I have 2 repositories: one working and one failing (both in the same computer with the same setup). They are public repositories.

[jsedanoj]

jsedano@riverwind MINGW64 ~/repos
$ cat ~/.gitconfig
[user]
        name = Javier Sedano
        email = javier.sedano@gmail.com

jsedano@riverwind MINGW64 ~/repos
$ git config --list
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
core.autocrlf=false
core.fscache=true
core.symlinks=false
pull.rebase=false
credential.helper=manager-core
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.name=Javier Sedano
user.email=javier.sedano@gmail.com

jsedano@riverwind MINGW64 ~/repos
$ 

[gagemillerlob]

Not sure what has changed here. Could you run the following commands locally and post the output here?

ssh -T git@github.com
git clone git@github.com:gagemillerlob/private-test.git
echo $env:SSH_AUTH_SOCK

I don't have github configured right now but here is the result from gitlab.

$ ssh -T git@gitlab.com
Welcome to GitLab, @gage.miller!

$ git clone git@gitlab.com:liveoak/ci-pipeline.git
Cloning into 'ci-pipeline'...
remote: Enumerating objects: 421, done.
remote: Counting objects: 100% (305/305), done.
remote: Compressing objects: 100% (290/290), done.
remote: Total 421 (delta 138), reused 4 (delta 4), pack-reused 116
Receiving objects: 100% (421/421), 80.63 KiB | 7.33 MiB/s, done.
Resolving deltas: 100% (178/178), done.
$ echo $env:SSH_AUTH_SOCK
:SSH_AUTH_SOCK

Running echo $env:SSH_AUTH_SOCK in powershell is also empty

[akutruff]

Any update here? This is a pretty nasty bug for anyone using dev containers. Thanks.

[chrmarti]

You all seem to be using MINGW. We are adding support for connecting to its ssh-agent only this milestone (currently fixing an issue in https://github.com/microsoft/vscode-remote-release/issues/7601), so I'm not sure why/how this ever used to work.

Or maybe you are using MINGW with Window's built-in ssh-agent (which uses a Windows socket path which we have been supporting for a long time)?

I'll let you know when the mentioned fix is in, chances are this affects the reports here too.

[akutruff]

Thanks for the update. How can I verify if vscode is using mingw or not?

[alshayed]

FWIW I'm using Git for Windows 2.38.1 with the Windows built-in openssh (I chose that option during Git install) and enabled the Windows SSH user agent in the services app.

[chrmarti]

The mentioned fix is released in Dev Containers 0.266.0-pre-release.

You can tell if you are using a MINGW (or Cygwin which is similar) ssh-agent by checking the dev container log after connecting to the dev container. If the local socket path looks like a file path, it's the MINGW ssh-agent, e.g.:

ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-898789e3c3ca3b1e95899334cc946fc07d1a0a72.sock) forwarded to local host (C:/Users/chrmarti/AppData/Local/Temp/ssh-Af2dCZ9pojS8/agent.221).

If the local socket path is a Windows socket path, you are using Windows' built-in ssh-agent:

ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-5bc51688929c6f38325d28261d210a97cfe963b8.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).

(It's always OpenSSH's ssh-agent, but with different socket path implementations.)

[jsedanoj]

Sorry, it does not work with 0.266.0-pre-release , it fails with the same error.

Here goes the log.

[42 ms] Dev Containers 0.266.0 in VS Code 1.73.1 (6261075646f055b99068d3688932416f2346dd3b).
[42 ms] Start: Resolving Remote
[52 ms] Start: Check Docker is running
[52 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[121 ms] Server API version: 1.41
[122 ms] Start: Run: docker volume ls -q
[182 ms] Start: Run: docker volume create --label vsch.local.repository=git@gitlab.com:javier-sedano/vaca.git --label vsch.local.repository.unique=false test-javi2
[251 ms] Start: Run: tar --no-same-owner -x -f -
[295 ms] Start: Run: docker build -f C:\Users\jsedano\AppData\Local\Temp\vsch\bootstrap-image\0.266.0\bootstrap.Dockerfile -t vsc-volume-bootstrap C:\Users\jsedano\AppData\Local\Temp\vsch\bootstrap-image\0.266.0
Sending build context to Docker daemon  2.199MB
Step 1/4 : FROM mcr.microsoft.com/devcontainers/base:0-alpine-3.16
 ---> 4120cfd0449e
Step 2/4 : RUN apk add --no-cache       nodejs  python3         npm     make   g
++      docker-cli      docker-cli-buildx       docker-cli-compose      ;
 ---> Using cache
 ---> 73f8e1d72722
Step 3/4 : RUN cd && npm i node-pty
 ---> Using cache
 ---> 31ae32001dff
Step 4/4 : COPY .vscode-remote-containers /root/.vscode-remote-containers
 ---> 0663333109de
Successfully built 0663333109de
Successfully tagged vsc-volume-bootstrap:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Win
dows Docker host. All files and directories added to build context will have '-r
wxr-xr-x' permissions. It is recommended to double check and reset permissions f
or sensitive files and directories.
[859 ms] Cloning Github repository: javier-sedano/vaca into /workspaces/vaca

[859 ms] Start: Run: docker run -d --mount type=volume,src=test-javi2,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[1279 ms] Start: Run in container: /bin/sh
[1293 ms] Start: Launching Dev Containers helper.
[1294 ms] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-7ce6ad7ed6defcb4b3bc9500e96c49d049ba69c0.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[1294 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[1309 ms] findLocalWindowsExecutable: Exectuable 'gpgconf' not found on PATH 'C:\Python310\Scripts\;C:\Python310\;C:\Program Files\Eclipse Adoptium\jdk-11.0.13.8-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Microsoft VS Code\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\dotnet\;C:\Program Files\nodejs\;C:\Users\jsedano\AppData\Local\Microsoft\WindowsApps;C:\Users\jsedano\AppData\Roaming\npm;C:\Users\jsedano\AppData\Local\Programs\Rancher Desktop\resources\resources\win32\bin;C:\Users\jsedano\AppData\Local\Programs\Rancher Desktop\resources\resources\linux\bin'.
[1312 ms] Start: Run in container: /bin/sh
[1327 ms] Start: Run in container: echo ~
[1488 ms] /root
[1488 ms] 
[1489 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-7ce6ad7ed6defcb4b3bc9500e96c49d049ba69c0.js' >/tmp/vscode-remote-containers-7ce6ad7ed6defcb4b3bc9500e96c49d049ba69c0.js
[1491 ms] 
[1491 ms] 
[1491 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-7ce6ad7ed6defcb4b3bc9500e96c49d049ba69c0.js' >/tmp/vscode-remote-containers-server-7ce6ad7ed6defcb4b3bc9500e96c49d049ba69c0.js_1670055563390
[1494 ms] 
[1494 ms] 
[1498 ms] Start: Run in container: # Test for /root/.gitconfig and git
[1500 ms] 
[1500 ms] 
[1500 ms] Start: Run in container: # Copy C:\Users\jsedano\.gitconfig to /root/.gitconfig
[1503 ms] 
[1503 ms] 
[1504 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global --replace-all credential.helper '!f() { node /tmp/vscode-remote-containers-7ce6ad7ed6defcb4b3bc9500e96c49d049ba69c0.js $*; }; f' || true
[1505 ms] 
[1506 ms] 
[1506 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[1508 ms] 
[1508 ms] 
[1508 ms] Start: Run in container: # Copy C:\Users\jsedano\.ssh\known_hosts to /root/.ssh/known_hosts
[1510 ms] 
[1511 ms] 
[1576 ms] Start: Run in container: git clone --depth 1 git@gitlab.com:javier-sedano/vaca.git .
Cloning into '.'...
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2448 ms] Start: Run: docker rm -f a8c529782ef9bcec30ba3b3a3f6570f3fde26e92f2116afe3a55d011fd7405b9
[2541 ms] Container server terminated (code: 137, signal: null).

[amurzeau]

I've tried to reproduce it as I've contributed to ssh-agent forwarding for Git Bash recently. But got the same error only when my github key is not in the SSH agent. Do you have you key in the SSH agent ?

When I do ssh-add $env:USERPROFILE\.ssh\github to add my key before cloning the repository in a container volume, the clone works fine. VS Code v1.74.0 and ms-vscode-remote.remote-containers v0.266.1.

Does ssh-add -l in powershell/cmd show your git key ?

When it fails to clone in the container, click on Edit devcontainer.json in Recovery Container. Then do ssh-add -l in a terminal, what do you get ?

[dallidalli]

My colleagues and me are facing the same issue/symptoms.

When using the current version, v0.266.1, (as well as the available pre-release, v0.267.0) of “Dev Containers” to “Clone in Container Volume”, the credentials, i.e. local ssh-keys, are not available in the container, resulting in a request for the (non-existing) git@…-password.

I do not get the option to Edit devcontainer.json in Recovery Container. Instead it halts completely at the password-prompt, where I'm unable to enter something. If I now close and reopen VSCode, I get the selection for which type of container I'd like to use. So there is no way for me to open the erronous container and run ssh-add -L. I've tried to do so via the CLI that Docker Desktop provides, but assume thats not the way to go, as independent of whether it's a container where cloning worked or not, I always receive Could not open a connection to your authentication agent. as a response.

Cloning the same repo outside of a Dev Container is possible, as well as selecting the option to “Create a Dev Container” (e.g. Ubuntu) instead of “Clone in Container Volume” via VSCode and afterwards cloning or checking ssh-add -L, everything looks fine. Same holds for any containers that are already set up properly, here the local ash-agent is also available.

Using version 0.255.4 of the Dev Containers Extension resolves the issue for us on Windows (10/11) using the built-in ssh.

EDIT: Added version numbers to the description above.

[jsedanoj]

I've tried to reproduce it as I've contributed to ssh-agent forwarding for Git Bash recently. But got the same error only when my github key is not in the SSH agent. Do you have you key in the SSH agent ?

When I do ssh-add $env:USERPROFILE\.ssh\github to add my key before cloning the repository in a container volume, the clone works fine. VS Code v1.74.0 and ms-vscode-remote.remote-containers v0.266.1.

Does ssh-add -l in powershell/cmd show your git key ?

When it fails to clone in the container, click on Edit devcontainer.json in Recovery Container. Then do ssh-add -l in a terminal, what do you get ?

Sorry, but this is not the reason. Not only I have the key in the agent (see screenshot below): with the same setup, in the same computer, I have one repo which clones correctly and another repo that fails; one version of the plugin works, another version fails. I'll copy paste here the same sumary I did in #7539 :

Up to now, I have made the following differential diagnose:

• This repository (and many others) works: https://gitlab.com/javier-sedano/kubos ; this repository fails: https://gitlab.com/javier-sedano/vaca . I have other failing repositories, but they are work-related repos not accesible by the public (the two I copied above are personal pulic repositories).
• The same repository works through HTTPS (https://gitlab.com/javier-sedano/vaca.git) but fails through SSH (git@gitlab.com:javier-sedano/vaca.git). Sounds sensible, since it is complaining about ssh public key. Cloning outside the container using Git Bash, works as expected.
• It fails with Windows, but works with Mac. Not tested in Linux.
• It fails both with Rancher Desktop for Windows and Docker Desktop for Windows (the later, not tested personally by me, but by one of my colleages).
• The work-repositories that are failing, are failing for several persons.
• The personal public repository above that is failing, is failing in all of my computers. The one that works, works in all of my computers.
• Downgrading the Dev Container extension to 0.255.4 solves the problem (but obviously fails again when upgrading to 0.262.3).

My guess is that... whatever is causing the problem in my vaca repo is not there neither in your test-case nor in my working kubos repo. Can you try with my failing vaca repo? This may be a hint...

[akutruff]

I verified that I'm not using MINGW and still getting the problem. I am trying this on pre-release version 0.267.0

ssh testing inside the recovery container:

0342548bdadb:/workspaces/blog# ssh-add -l
4096 SHA256:<redacted> <redacted> (RSA)
256 SHA256:<redacted> <redacted> (ED25519)
0342548bdadb:/workspaces/blog# ssh -T git@github.com
git@github.com: Permission denied (publickey).

.gitconfig

[user]
    email = 874049+akutruff@users.noreply.github.com
    name = akutruff
[winUpdater]
    recentlySeenVersion = 2.21.0.windows.1

Docker log:

[2022-12-09T16:35:13.398Z] Start: Run: docker run -d --mount type=volume,src=blog,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[2022-12-09T16:35:14.085Z] Stop (687 ms): Run: docker run -d --mount type=volume,src=blog,dst=/workspaces -v /var/run/docker.sock:/var/run/docker.sock vsc-volume-bootstrap sleep infinity
[2022-12-09T16:35:14.086Z] Start: Run in container: /bin/sh
[2022-12-09T16:35:14.113Z] Start: Launching Dev Containers helper.
[2022-12-09T16:35:14.113Z] ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-6b2b2c6b752735f23c299467986f76b199b47019.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
[2022-12-09T16:35:14.113Z] Start: Run: gpgconf --list-dir agent-extra-socket
[2022-12-09T16:35:14.165Z] Stop (52 ms): Run: gpgconf --list-dir agent-extra-socket
[2022-12-09T16:35:14.165Z] C:\Users\andyk\AppData\Roaming\gnupg\S.gpg-agent.extra
[2022-12-09T16:35:14.165Z] 
[2022-12-09T16:35:14.165Z] Start: Run in container: gpgconf --list-dir agent-socket
[2022-12-09T16:35:14.359Z] /root/.gnupg/S.gpg-agent
[2022-12-09T16:35:14.359Z] 
[2022-12-09T16:35:14.359Z] Stop (194 ms): Run in container: gpgconf --list-dir agent-socket
[2022-12-09T16:35:14.359Z] Start: Run in container: gpgconf --list-dir homedir
[2022-12-09T16:35:14.362Z] /root/.gnupg
[2022-12-09T16:35:14.362Z] 
[2022-12-09T16:35:14.362Z] Stop (3 ms): Run in container: gpgconf --list-dir homedir
[2022-12-09T16:35:14.362Z] Start: Run in container: ls '/root/.gnupg/private-keys-v1.d' 2>/dev/null
[2022-12-09T16:35:14.365Z] 
[2022-12-09T16:35:14.365Z] 
[2022-12-09T16:35:14.365Z] Exit code 2
[2022-12-09T16:35:14.365Z] Stop (3 ms): Run in container: ls '/root/.gnupg/private-keys-v1.d' 2>/dev/null
[2022-12-09T16:35:14.366Z] Start: Run in container: mkdir -p -m 700 '/root/.gnupg'
[2022-12-09T16:35:14.366Z] Start: Run in container: /bin/sh
[2022-12-09T16:35:14.367Z] Stop (254 ms): Launching Dev Containers helper.
[2022-12-09T16:35:14.368Z] 
[2022-12-09T16:35:14.369Z] 
[2022-12-09T16:35:14.369Z] Stop (3 ms): Run in container: mkdir -p -m 700 '/root/.gnupg'
[2022-12-09T16:35:14.369Z] Start: Run: gpgconf --list-dir homedir
[2022-12-09T16:35:14.398Z] Start: Run in container: echo ~
[2022-12-09T16:35:14.416Z] Stop (47 ms): Run: gpgconf --list-dir homedir
[2022-12-09T16:35:14.416Z] C:\Users\andyk\AppData\Roaming\gnupg
[2022-12-09T16:35:14.416Z] 
[2022-12-09T16:35:14.416Z] Start: Run in container: gpgconf --list-dir homedir
[2022-12-09T16:35:14.419Z] /root/.gnupg
[2022-12-09T16:35:14.419Z] 
[2022-12-09T16:35:14.419Z] Stop (3 ms): Run in container: gpgconf --list-dir homedir
[2022-12-09T16:35:14.419Z] Start: Run in container: # Test for /root/.gnupg/pubring.kbx and gpg
[2022-12-09T16:35:14.422Z] 
[2022-12-09T16:35:14.422Z] 
[2022-12-09T16:35:14.422Z] Stop (3 ms): Run in container: # Test for /root/.gnupg/pubring.kbx and gpg
[2022-12-09T16:35:14.422Z] Start: Run in container: # Copy C:\Users\andyk\AppData\Roaming\gnupg\pubring.kbx to /root/.gnupg/pubring.kbx
[2022-12-09T16:35:14.425Z] 
[2022-12-09T16:35:14.426Z] 
[2022-12-09T16:35:14.426Z] Stop (4 ms): Run in container: # Copy C:\Users\andyk\AppData\Roaming\gnupg\pubring.kbx to /root/.gnupg/pubring.kbx
[2022-12-09T16:35:14.427Z] Start: Run in container: # Test for /root/.gnupg/trustdb.gpg and gpg
[2022-12-09T16:35:14.429Z] 
[2022-12-09T16:35:14.429Z] 
[2022-12-09T16:35:14.429Z] Stop (2 ms): Run in container: # Test for /root/.gnupg/trustdb.gpg and gpg
[2022-12-09T16:35:14.429Z] Start: Run in container: # Copy C:\Users\andyk\AppData\Roaming\gnupg\trustdb.gpg to /root/.gnupg/trustdb.gpg
[2022-12-09T16:35:14.432Z] 
[2022-12-09T16:35:14.432Z] 
[2022-12-09T16:35:14.433Z] Stop (4 ms): Run in container: # Copy C:\Users\andyk\AppData\Roaming\gnupg\trustdb.gpg to /root/.gnupg/trustdb.gpg
[2022-12-09T16:35:14.433Z] Start: Run: gpg-connect-agent updatestartuptty /bye
[2022-12-09T16:35:14.480Z] Stop (47 ms): Run: gpg-connect-agent updatestartuptty /bye
[2022-12-09T16:35:14.683Z] /root
[2022-12-09T16:35:14.683Z] 
[2022-12-09T16:35:14.683Z] Stop (285 ms): Run in container: echo ~
[2022-12-09T16:35:14.683Z] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-6b2b2c6b752735f23c299467986f76b199b47019.js' >/tmp/vscode-remote-containers-6b2b2c6b752735f23c299467986f76b199b47019.js
[2022-12-09T16:35:14.686Z] 
[2022-12-09T16:35:14.686Z] 
[2022-12-09T16:35:14.686Z] Stop (3 ms): Run in container: cat <<'EOF-/tmp/vscode-remote-containers-6b2b2c6b752735f23c299467986f76b199b47019.js' >/tmp/vscode-remote-containers-6b2b2c6b752735f23c299467986f76b199b47019.js
[2022-12-09T16:35:14.686Z] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-6b2b2c6b752735f23c299467986f76b199b47019.js' >/tmp/vscode-remote-containers-server-6b2b2c6b752735f23c299467986f76b199b47019.js_1670603714686
[2022-12-09T16:35:14.690Z] 
[2022-12-09T16:35:14.691Z] 
[2022-12-09T16:35:14.691Z] Stop (5 ms): Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-6b2b2c6b752735f23c299467986f76b199b47019.js' >/tmp/vscode-remote-containers-server-6b2b2c6b752735f23c299467986f76b199b47019.js_1670603714686
[2022-12-09T16:35:14.693Z] Start: Run in container: # Test for /root/.gitconfig and git
[2022-12-09T16:35:14.695Z] 
[2022-12-09T16:35:14.696Z] 
[2022-12-09T16:35:14.696Z] Stop (3 ms): Run in container: # Test for /root/.gitconfig and git
[2022-12-09T16:35:14.696Z] Start: Run in container: # Copy C:\Users\andyk\.gitconfig to /root/.gitconfig
[2022-12-09T16:35:14.699Z] 
[2022-12-09T16:35:14.699Z] 
[2022-12-09T16:35:14.699Z] Stop (3 ms): Run in container: # Copy C:\Users\andyk\.gitconfig to /root/.gitconfig
[2022-12-09T16:35:14.699Z] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global --replace-all credential.helper '!f() { node /tmp/vscode-remote-containers-6b2b2c6b752735f23c299467986f76b199b47019.js $*; }; f' || true
[2022-12-09T16:35:14.702Z] 
[2022-12-09T16:35:14.702Z] 
[2022-12-09T16:35:14.702Z] Stop (3 ms): Run in container: command -v git >/dev/null 2>&1 && git config --global --replace-all credential.helper '!f() { node /tmp/vscode-remote-containers-6b2b2c6b752735f23c299467986f76b199b47019.js $*; }; f' || true
[2022-12-09T16:35:14.702Z] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[2022-12-09T16:35:14.705Z] 
[2022-12-09T16:35:14.705Z] 
[2022-12-09T16:35:14.705Z] Stop (3 ms): Run in container: # Test for /root/.ssh/known_hosts and ssh
[2022-12-09T16:35:14.705Z] Start: Run in container: # Copy C:\Users\andyk\.ssh\known_hosts to /root/.ssh/known_hosts
[2022-12-09T16:35:14.709Z] 
[2022-12-09T16:35:14.709Z] 
[2022-12-09T16:35:14.709Z] Stop (4 ms): Run in container: # Copy C:\Users\andyk\.ssh\known_hosts to /root/.ssh/known_hosts
[2022-12-09T16:35:14.765Z] Start: Run in container: git clone --depth 1 git@github.com:akutruff/blog.git .
[2022-12-09T16:35:14.890Z] 
[2022-12-09T16:35:15.085Z] Cloning into '.'...
[2022-12-09T16:35:15.246Z] git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2022-12-09T16:35:15.339Z] Stop (574 ms): Run in container: git clone --depth 1 git@github.com:akutruff/blog.git .
[2022-12-09T16:35:15.340Z] Start: Run: docker rm -f 37eb005cc85b22e442aaa3ae87732ab2a12f452ad64ae6ca24b6df5546077434
[2022-12-09T16:35:15.516Z] Stop (1430 ms): Run in container: /bin/sh
[2022-12-09T16:35:15.535Z] Stop (1169 ms): Run in container: /bin/sh
[2022-12-09T16:35:15.535Z] Container server terminated (code: 137, signal: null).
[2022-12-09T16:35:15.878Z] Stop (538 ms): Run: docker rm -f 37eb005cc85b22e442aaa3ae87732ab2a12f452ad64ae6ca24b6df5546077434
[2022-12-09T16:35:28.013Z] Start: Run: docker version --format {{.Server.APIVersion}}
[2022-12-09T16:35:28.166Z] Stop (153 ms): Run: docker version --format {{.Server.APIVersion}}
[2022-12-09T16:35:28.166Z] 1.41

[amurzeau]

I verified that I'm not using MINGW and still getting the problem. I am trying this on pre-release version 0.267.0

ssh testing inside the recovery container:

0342548bdadb:/workspaces/blog# ssh-add -l
4096 SHA256:<redacted> <redacted> (RSA)
256 SHA256:<redacted> <redacted> (ED25519)
0342548bdadb:/workspaces/blog# ssh -T git@github.com
git@github.com: Permission denied (publickey).

So your github key is one of either RSA or ED25519 keys, but ssh still refuse to login to github ?

Is there any clue about why ssh fails in verbose logs using ssh -T -v git@github.com in the recovery container ?

ssh client changed from v8.6p1 to v9.0p1 between ms-vscode-remote.remote-containers v0.256.0 and v0.259.0 (according to bootstrap.Dockerfile changes on available versions on vsixhub.com). Maybe that's related to the issue, but I didn't find what by reading the openssh changelog.

Alternatively, finding the exact version that introduced the regression can also help (by trying every single version since 0.255.4).

[jsedanoj]

Beware: the last working version for me is v0.255.4, not v0.256.0. Actually, v0.256.0 is not even offered to me in the dropdown (see screenshot below).

Also, notice that in my case, with the same version v0.262.3, some of the repos work and some others fail (see https://github.com/microsoft/vscode-remote-release/issues/7482#issuecomment-1343347658). They are public repos, you can try to clone them and see if the problem is also happening to you (whether if it happens or not, it may be a clue).

[amurzeau]

I've tried to clone git@gitlab.com:javier-sedano/vaca.git using:

• a RSA 4096 bits SHA256 key for gitlab access
• Docker Desktop 20.10.14
• Windows 10 10.0.19044.1706
• ssh-agent service from Windows 10 (OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2)
• VSCode 1.72.2
• Dev Containers extension 0.266.1

And reproduced it :)

For a fast-forward/TLDR to solution, go to the end of this message.

I've tried before but not with the Windows 10 bundled ssh-agent service (using a ssh-agent "forwarder" to pageant) and it was working fine. So the issue is a combination of recent Dev Containers extension with the ssh-agent service.

Here is the analysis of the issue step by step:

There is nothing appearing in the Output logs. When reopening in a recovery container, doing ssh -Tv git@gitlab.com output this:

[...]
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
[...]

But ssh-add -l lists the keys correctly.

Using a combination of socat and tcpdump, I was able to dump the exchanged data between ssh-agent and ssh inside the container, here is the dump:

ssh-agent tcpdump

``` a6a44b4ea9cb:/workspaces/vaca# ssh -T git@gitlab.com 18:15:53.746225 IP localhost.35156 > localhost.6000: Flags [S], seq 1390537246, win 65495, options [mss 65495,sackOK,TS val 3741389040 ecr 0,nop,wscale 7], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 003c 8956 4000 4006 b363 7f00 0001 7f00 .<.V@.@..c...... 0x0020: 0001 8954 1770 52e1 ea1e 0000 0000 a002 ...T.pR......... 0x0030: ffd7 fe30 0000 0204 ffd7 0402 080a df01 ...0............ 0x0040: 10f0 0000 0000 0103 0307 .......... 18:15:53.746240 IP localhost.6000 > localhost.35156: Flags [S.], seq 1520255142, ack 1390537247, win 65483, options [mss 65495,sackOK,TS val 3741389040 ecr 3741389040,nop,wscale 7], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 003c 0000 4000 4006 3cba 7f00 0001 7f00 .<..@.@.<....... 0x0020: 0001 1770 8954 5a9d 40a6 52e1 ea1f a012 ...p.TZ.@.R..... 0x0030: ffcb fe30 0000 0204 ffd7 0402 080a df01 ...0............ 0x0040: 10f0 df01 10f0 0103 0307 .......... 18:15:53.746253 IP localhost.35156 > localhost.6000: Flags [.], ack 1, win 512, options [nop,nop,TS val 3741389040 ecr 3741389040], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 8957 4000 4006 b36a 7f00 0001 7f00 .4.W@.@..j...... 0x0020: 0001 8954 1770 52e1 ea1f 5a9d 40a7 8010 ...T.pR...Z.@... 0x0030: 0200 fe28 0000 0101 080a df01 10f0 df01 ...(............ 0x0040: 10f0 .. 18:15:53.746483 IP localhost.35156 > localhost.6000: Flags [P.], seq 1:282, ack 1, win 512, options [nop,nop,TS val 3741389040 ecr 3741389040], length 281 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 014d 8958 4000 4006 b250 7f00 0001 7f00 .M.X@.@..P...... 0x0020: 0001 8954 1770 52e1 ea1f 5a9d 40a7 8018 ...T.pR...Z.@... 0x0030: 0200 ff41 0000 0101 080a df01 10f0 df01 ...A............ 0x0040: 10f0 0000 0115 1b00 0000 1873 6573 7369 ...........sessi 0x0050: 6f6e 2d62 696e 6440 6f70 656e 7373 682e on-bind@openssh. 0x0060: 636f 6d00 0000 6800 0000 1365 6364 7361 com...h....ecdsa 0x0070: 2d73 6861 322d 6e69 7374 7032 3536 0000 -sha2-nistp256.. 0x0080: 0008 6e69 7374 7032 3536 0000 0041 0454 ..nistp256...A.T 0x0090: 8cab 325e 57da d4cd 4e24 5a2b 468d e478 ..2^W...N$Z+F..x 0x00a0: 3d64 9adb d48f a89d 5f56 486d 8f71 f0f7 =d......_VHm.q.. 0x00b0: 18cb 5575 58ad 4f50 9585 ab38 b5c1 a50e ..UuX.OP...8.... 0x00c0: 8a5a 36e3 4c34 dd0e 4a28 845c e06c 4600 .Z6.L4..J(.\.lF. 0x00d0: 0000 20b8 38da 6e25 2e17 e074 bbe4 3477 ....8.n%...t..4w 0x00e0: 3c50 4649 0c31 05c4 cf96 8d45 1bc8 0965 localhost.35156: Flags [.], ack 282, win 510, options [nop,nop,TS val 3741389040 ecr 3741389040], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 89d3 4000 4006 b2ee 7f00 0001 7f00 .4..@.@......... 0x0020: 0001 1770 8954 5a9d 40a7 52e1 eb38 8010 ...p.TZ.@.R..8.. 0x0030: 01fe fe28 0000 0101 080a df01 10f0 df01 ...(............ 0x0040: 10f0 .. 2022/12/11 18:15:53 socat[2110] E write(5, 0x7fc37f73f000, 5): Broken pipe 18:15:53.823148 IP localhost.6000 > localhost.35156: Flags [F.], seq 1, ack 282, win 512, options [nop,nop,TS val 3741389116 ecr 3741389040], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 89d4 4000 4006 b2ed 7f00 0001 7f00 .4..@.@......... 0x0020: 0001 1770 8954 5a9d 40a7 52e1 eb38 8011 ...p.TZ.@.R..8.. 0x0030: 0200 fe28 0000 0101 080a df01 113c df01 ...(.........<.. 0x0040: 10f0 .. 18:15:53.823299 IP localhost.35156 > localhost.6000: Flags [.], ack 2, win 512, options [nop,nop,TS val 3741389117 ecr 3741389116], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 8959 4000 4006 b368 7f00 0001 7f00 .4.Y@.@..h...... 0x0020: 0001 8954 1770 52e1 eb38 5a9d 40a8 8010 ...T.pR..8Z.@... 0x0030: 0200 fe28 0000 0101 080a df01 113d df01 ...(.........=.. 0x0040: 113c .< 18:15:53.823777 IP localhost.35156 > localhost.6000: Flags [P.], seq 282:287, ack 2, win 512, options [nop,nop,TS val 3741389117 ecr 3741389116], length 5 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0039 895a 4000 4006 b362 7f00 0001 7f00 .9.Z@.@..b...... 0x0020: 0001 8954 1770 52e1 eb38 5a9d 40a8 8018 ...T.pR..8Z.@... 0x0030: 0200 fe2d 0000 0101 080a df01 113d df01 ...-.........=.. 0x0040: 113c 0000 0001 0b .<..... 18:15:53.823787 IP localhost.6000 > localhost.35156: Flags [.], ack 287, win 512, options [nop,nop,TS val 3741389117 ecr 3741389117], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 89d5 4000 4006 b2ec 7f00 0001 7f00 .4..@.@......... 0x0020: 0001 1770 8954 5a9d 40a8 52e1 eb3d 8010 ...p.TZ.@.R..=.. 0x0030: 0200 fe28 0000 0101 080a df01 113d df01 ...(.........=.. 0x0040: 113d .= 18:15:53.823808 IP localhost.35156 > localhost.6000: Flags [F.], seq 287, ack 2, win 512, options [nop,nop,TS val 3741389117 ecr 3741389117], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 895b 4000 4006 b366 7f00 0001 7f00 .4.[@.@..f...... 0x0020: 0001 8954 1770 52e1 eb3d 5a9d 40a8 8011 ...T.pR..=Z.@... 0x0030: 0200 fe28 0000 0101 080a df01 113d df01 ...(.........=.. 0x0040: 113d .= 18:15:53.823818 IP localhost.6000 > localhost.35156: Flags [.], ack 288, win 512, options [nop,nop,TS val 3741389117 ecr 3741389117], length 0 0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 ..............E. 0x0010: 0034 89d6 4000 4006 b2eb 7f00 0001 7f00 .4..@.@......... 0x0020: 0001 1770 8954 5a9d 40a8 52e1 eb3e 8010 ...p.TZ.@.R..>.. 0x0030: 0200 fe28 0000 0101 080a df01 113d df01 ...(.........=.. 0x0040: 113d .= ```

The connection is closed by VSCode ssh-agent forwarder after the first packet sent by ssh, which contains session-bind@openssh.com message. The opcode of this message is 0x1B (27 in decimal) meaning this is a SSH_AGENTC_EXTENSION message.

When running Windows' ssh-agent in debug mode and then retrying to connect to gitlab in the container, it shows why it closes the connection:

C:\ProgramData\ssh>ssh-agent -d
agent_start pid:1560, dbg:1
client pid 4964 connected
agent_process_connection pipe:000000000000014C
debug1: client type: restricted user
debug1: process agent request type 27
debug1: unknown agent request 27
debug1: connection 00000224A2C81DF0 clean up
debug1: iocp error: 6 on 0000000000000000

Request type 27 is SSH_AGENTC_EXTENSION.

Windows' OpenSSH version is 8.1p1, and this message was introduced in OpenSSH 8.9.1.0p1. So it means that: Containers' ssh client is trying to use an unsupported feature of Windows' ssh-agent.

This issue was also reported outside the use of VScode, for example:

• https://stackoverflow.com/questions/71399334/ssh-fails-to-use-private-key-from-ssh-agent-communication-with-agent-failed
• https://github.com/PowerShell/Win32-OpenSSH/issues/1958

---

I see these solutions:

• Upgrade Windows' ssh-agent to a more recent one, assuming it exists and support the new SSH_AGENTC_EXTENSION feature. This feature was implemented in OpenSSH version v8.9.0.0.

• Workaround the issue by using a different ssh-agent, with a ssh-agent forwarder if needed

• Downgrade the container's ssh version to <= 8.8 or less

• Make VSCode automatically reconnect to the ssh-agent's pipe if the connection is closed (while keeping the connection on ssh side open). This allows to handle the error more gracefully (by replying an error message instead of closing the connection). The sequence of messages would be:

  • ssh client sends the SSH_AGENTC_EXTENSION message
  • VSCode forwards it to ssh-agent
  • ssh-agent does not support this message and close the connection
  • VSCode sends back to ssh client a basic error reply without closing the connection (bytes [0x00, 0x00, 0x00, 0x01, 0x05])
  • VSCode reopens the connection to ssh-agent
  • ssh client sends other requests
  • VSCode forwards them to the reopened ssh-agent connection

• The issue I've reproduced here is maybe not the issue of the authors, and there is something else to look for (but I hope not lol)

Anyway, upgrading OpenSSH by installing the newest available, at least v8.9.0.0) should fix the issue, I've tested V8.9.1.0p1-Beta, and it fixed the issue for me, cloning in a container volume was working again.

[jsedanoj]

I've tried to clone git@gitlab.com:javier-sedano/vaca.git using:

• a RSA 4096 bits SHA256 key for gitlab access
• Docker Desktop 20.10.14
• Windows 10 10.0.19044.1706
• ssh-agent service from Windows 10 (OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2)
• VSCode 1.72.2
• Dev Containers extension 0.266.1

And reproduced it :)

Good to know!

However, I do not fully understand if my environment is your environment or how to check it. For ssh-agent, I am using... I do not know... the one which comes with Windows 10 (Settings --> Apps --> Optional Features --> OpenSSH Client (see screenshot).

How can I check the version I am using? It is the version that comes with Windows 10.

Are you asking to install one of these beta versions? https://github.com/PowerShell/Win32-OpenSSH/releases

My opinion: if the extension does not work with the default version installed by Windows, we are losing a big share of the cake.

Also... take into consideration that in my case cloning git@gitlab.com:javier-sedano/vaca.git fails but cloning git@gitlab.com:javier-sedano/kubos.git ... any idea why? Have you tried cloing both?

[akutruff]

So your github key is one of either RSA or ED25519 keys, but ssh still refuse to login to github ?

Is there any clue about why ssh fails in verbose logs using ssh -T -v git@github.com in the recovery container ?

interestingly it seems its looking for ssh keys in root and failing to find them, even though ssh-add -l finds my keys just fine.

0342548bdadb:/workspaces/blog# ssh-add -l
4096 SHA256:<redacted> <redacted> (RSA)
256 SHA256:<redacted> main (ED25519)

0342548bdadb:/workspaces/blog# ls -al ~/.ssh
total 20
drwxr-xr-x 2 root root 4096 Dec  9 16:24 .
drwx------ 1 root root 4096 Dec 12 18:50 ..
-rw-r--r-- 1 root root 8563 Dec  9 16:24 known_hosts

0342548bdadb:/workspaces/blog# ssh -T -v git@github.com
OpenSSH_9.0p1, OpenSSL 1.1.1s  1 Nov 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to github.com [140.82.114.4] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version babeld-181fb29f
debug1: compat_banner: no match: babeld-181fb29f
debug1: Authenticating to github.com:22 as 'git'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'github.com' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: ssh_fetch_identitylist: communication with agent failed
debug1: Will attempt key: /root/.ssh/id_rsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa 
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /root/.ssh/id_ed25519 
debug1: Will attempt key: /root/.ssh/id_ed25519_sk 
debug1: Will attempt key: /root/.ssh/id_xmss 
debug1: Will attempt key: /root/.ssh/id_dsa 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug1: Trying private key: /root/.ssh/id_xmss
debug1: Trying private key: /root/.ssh/id_dsa
debug1: No more authentication methods to try.
git@github.com: Permission denied (publickey).

[amurzeau]

@jsedanoj

In the services list, open the properties dialog for OpenSSH Authentication Agent. Then check where the executable is stored, this should probably be in C:\Windows\System32\OpenSSH (unless you installed it via provided installers on github).

Then run ssh -V in the same folder as ssh-agent service executable, for example: C:\Windows\System32\OpenSSH\ssh.exe -V

Yes I'm proposing to try the beta versions, along with other solutions. You can also try to use MinGW's ssh-agent too. The issue is that openssh has a breaking change in the ssh-agent protocol by adding SSH_AGENTC_EXTENSION.

I've no idea why it would work with one repo but not the other one. Especially considering ssh will send you key before git even has the opportunity to tell which repo to clone (I'm not sure about that though, I don't know how this work). But I've not tried the other one (didn't have enough time to check both)

@akutruff You get get_agent_identities: ssh_fetch_identitylist: communication with agent failed too (same as me), so I guess you have the same issue: too old ssh-agent on Windows.

Another workaround, which is more of a hack, is to change the extensions/ms-vscode-remote.remote-containers-0.266.1/scripts/bootstrap.Dockerfile file and replace mcr.microsoft.com/devcontainers/base:0-alpine-3.16 with mcr.microsoft.com/devcontainers/base:0-alpine-3.14 which contains ssh 8.6 instead of ssh 9.0 and should also fix the issue (until the next extension update).

[jsedanoj]

Thanks. I can confirm I have your very same version:

C:\Users\jsedano>C:\Windows\System32\OpenSSH\ssh.exe -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

C:\Users\jsedano>

Looks like we are in a bad situation then.

• Upgrading OpenSSH to a beta version that has not evolved since Spring 2022... well... it is more than acceptable for me in my personal projects, but will be hard to explain in corporate environments.
• Using a different ssh-agent... well... for me it violates the very core benefit of the dev-container environment. The main benefit, for me, is: install VSCode + dev-container, DockerDesktop/RancherDesktop... and that's all: the full set of development tools are inside the Docker, defined in the .devcontainer. The more tools you need to install, the farther we are from this scenario. Until now, OpenSSH was the official OpenSSH by MS. Very easy to defend against, for example, the Security department or the Licenses Procurement department.
• Downgrade the container ssh version... not only it looks like a hack, it also implies using a non-latest version; so in a few months it will be obsolete.

Not blaming, not at all. I appreciate the work. Just explaining my view of the Cons.

:'(

[echatzikyriakidis]

Any news on that? Is it going to be solved soon? Would like to ask, which is the latest version of Dev Containers extension where it can clone private repo with SSH without problems?

[jsedano-emobg]

Any news on that? Is it going to be solved soon? Would like to ask, which is the latest version of Dev Containers extension where it can clone private repo with SSH without problems?

0.255.4

See https://github.com/microsoft/vscode-remote-release/issues/7482#issuecomment-1345255725

[chrmarti]

Thanks @amurzeau for the detailed analysis! I'm downgrading the OpenSSH client we use when cloning to version 8.8 as a fix.

That fix is available in Dev Containers v0.268.0-pre-release. Could someone give that a try and let me know if that fixes the issue? Thanks!

[jsedanoj]

I can confirm v0.268.0-pre-release works with my repository https://gitlab.com/javier-sedano/vaca (which was failing before).

[chrmarti]

Thanks everyone, considering this fixed. We will track the more general issue in https://github.com/microsoft/vscode-remote-release/issues/7525.

[htinaunglu]

Thanks, with the dev container pre-release v0..268.0 solved the problem!