search

microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.69k stars 297 forks source link

Using a key-pair with a passphrase, unable to clone private repo to dev container #9163

Open pjmattingly opened 1 year ago

pjmattingly commented 1 year ago

Type: Bug

1) Configure github for public/private key access 2) Configure the key with a passphrase 3) In VS Code, Open command palette 4) enter: Dev Container: Clone Repository in Container Volume 5) enter name of private repo (i.e., pjmattingly/obi-wan) that is secured by the key 4) Observe no response

Looking at the Dev Container console (?) the following output is shown:

[919098 ms] Start: Run: wsl -d Ubuntu2 -e wslpath -u \\wsl.localhost\Ubuntu2\home\peter\Canonical\TODO, tracking\notes
[919214 ms] Start: Run: wsl -d Ubuntu2 -e /bin/sh -c cd '/home/peter/Canonical/TODO, tracking/notes' && /bin/sh
[919217 ms] Start: Run in host: id -un
[919278 ms] peter
[919278 ms] 
[919278 ms] Start: Run in host:  (command -v getent >/dev/null 2>&1 && getent passwd 'peter' || grep -E '^peter|^[^:]*:[^:]*:peter:' /etc/passwd || true)
[919280 ms] Start: Run in host: echo ~
[919280 ms] /home/peter
[919280 ms] 
[919281 ms] Start: Run in host: test -x '/home/peter/.vscode-remote-containers/bin/d037ac076cee195194f93ce6fe2bdfe2969cc82d/node'
[919281 ms] 
[919281 ms] 
[919281 ms] Start: Run in host: test -f '/home/peter/.vscode-remote-containers/dist/vscode-remote-containers-server-0.321.0.js'
[919282 ms] 
[919282 ms] 
[919282 ms] userEnvProbe: loginInteractiveShell (default)
[919282 ms] userEnvProbe: not found in cache
[919282 ms] userEnvProbe shell: /bin/bash
[919591 ms] userEnvProbe PATHs:
Probe:     '/home/peter/.nvm/versions/node/v18.16.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files (x86)/VMware/VMware Player/bin/:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/PowerShell/7/:/Docker/host/bin:/mnt/c/Program Files/GitHub CLI/:/mnt/c/cmder:/mnt/c/Users/peter-work/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/peter-work/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/peter-work/AppData/Local/GitHubDesktop/bin:/mnt/c/Program Files/GitHub CLI:/snap/bin:/home/peter/.local/bin:./node_modules/.bin'
Container: None
[919635 ms] Start: Run in Host: docker version --format {{.Server.APIVersion}}
[919682 ms] 1.43
[919635 ms] Start: Run in Host: git ls-remote https://github.com/pjmattingly/obi-wan

Which then hangs with no response (i.e., after 10+ minutes of waiting with no response); Perhaps it's waiting to resolve a passphrase prompt that is never shown?

Attempting to clone with ssh is more instructive:

<snip>

[1165988 ms] Start: Run in Host: git ls-remote git@github.com:pjmattingly/obi-wan.git
[1166664 ms] 
[1166664 ms] ssh_askpass: exec(c:\\Users\\peter-work\\.vscode\\extensions\\ms-vscode-remote.remote-containers-0.321.0\\scripts\\ssh-askpass.bat): No such file or directory
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[1166664 ms] Exit code 128

Note that this command completes correctly in a VS Code terminal:

git ls-remote git@github.com:pjmattingly/obi-wan.git
3cbad27c4a46366e955c735593f365552a52d7e1        HEAD
039a86729b4af8bf7e2ecdd52a7b5689fb3a3316        <redacted>
eeb80ba49ed7fc55b0aa46346b288592094ce758        <redacted>
bf245fc86aa9fb6f8f41a63fc5dd2a179f049734        <redacted>

<snip>

(note that branch names have been redacted)

Then also, cloning the repo via git without ssh also works correctly.

I had followed this guide to setup the ssh-agent and register my key-pair: https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement.

Then also, to get VS-Code to correctly query the running ssh-agent, the environment variable GIT_SSH was set (GIT_SSH=C:\WINDOWS\System32\OpenSSH\ssh.exe); See these discussions:

https://groups.google.com/g/git-for-windows/c/eCH80oJTL9A?pli=1 https://github.com/PowerShell/Win32-OpenSSH/issues/1136#issuecomment-382671392 https://github.com/PowerShell/Win32-OpenSSH/wiki/Setting-up-a-Git-server-on-Windows-using-Git-for-Windows-and-Win32_OpenSSH

So, it seems as if the Dev Containers addon is not properly querying the running ssh-agent and is likely not parsing environment variables related to ssh and git.

Extension version: 0.321.0 VS Code version: Code 1.84.0 (d037ac076cee195194f93ce6fe2bdfe2969cc82d, 2023-11-01T11:29:04.398Z) OS version: Windows_NT x64 10.0.22621 Modes:

System Info |Item|Value| |---|---| |CPUs|12th Gen Intel(R) Core(TM) i7-12700K (20 x 3610)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|undefined| |Memory (System)|63.79GB (33.40GB free)| |Process Argv|--crash-reporter-id 131765d7-dc5e-4dfb-b6ae-912ece1e4ed1| |Screen Reader|no| |VM|40%|
A/B Experiments ``` vsliv368:30146709 vsreu685:30147344 python383:30185418 vspor879:30202332 vspor708:30202333 vspor363:30204092 vslsvsres303:30308271 vserr242:30382549 pythontb:30283811 vsjup518:30340749 pythonptprofiler:30281270 vshan820:30294714 vstes263:30335439 vscoreces:30445986 vscod805cf:30301675 binariesv615:30325510 bridge0708:30335490 bridge0723:30353136 vsaa593:30376534 pythonvs932:30410667 py29gd2263cf:30856253 vscaat:30438848 vsclangdf:30486550 c4g48928:30535728 dsvsc012:30540252 pynewext54:30695312 azure-dev_surveyone:30548225 3biah626:30602489 f6dab269:30613381 a9j8j154:30646983 showlangstatbar:30737416 a2ce3375:30757347 pythonfmttext:30731395 fixshowwlkth:30771522 showindicator:30805244 pythongtdpath:30769146 i26e3531:30792625 pythonnosmt12:30797651 pythonidxpt:30866567 pythonnoceb:30805159 synctok:30869157 dsvsc013:30795093 dsvsc014:30804076 dsvsc015:30845448 pythontestfixtcf:30871695 pythonregdiag2:30871582 pyreplss2:30879913 pythonmypyd1:30879173 pythoncet00cf:30874137 pythontbext0:30879054 ```
JOSEPHGILBY commented 10 months ago

@chrmarti, any chance on getting prioritization / milestone indication for this issue? There are a couple of other people who are running into similar (or likely the same issue) and it appears that this gets in the way of using the "clone repository into container volume" on Windows for private repos.

kiweezi commented 8 months ago

Is there a workaround for this, or any comments? Currently it blocks my whole team and I've spent countless hours trying everything to get around it.

It's essential for us as we use Windows. There is a bug in WSL which means git runs extremely slowly when using a dev container without cloning to the container volume. This same bug has been open since 2019, so looks like it will never be worked on.

Currently Windows users cannot use the "clone repository into container volume" feature on private repos. That in combination with the WSL bug, has likely prevented most Windows users (like us) from running dev containers entirely.

As @JOSEPHGILBY says, it's a bit strange that this has no prioritization. It's the most common workflow for a Windows developer using dev containers, so likely effects most Windows users. With the lack of any errors, new users have likely turned away already.

Bajchos commented 8 months ago

@kiweezi I had the same problem and it seems that I have found workaround.

Use WSL vscode extension and command (CTRL+SHIFT+P) WSL: Connect to WSL. Open Explorer (CTRL+SHIFT+E) and press Clone Repository blue button. Follow quick wizard: Repository source -> Repository name -> local folder where to clone Open Remote Explorer extension and click Open Folder in Container or use command (CTRL+SHIFT+P) Dev Containers: Open Folder in Container... Target folder where you cloned it in WSL. That should be it, your cloned repository should be running inside container now (I haven't noticed any problems with performance so far).

Hope this helps.

kiweezi commented 8 months ago

@Bajchos you're a saviour! That's a great workaround and super easy to follow steps. It's working as we need now!

Thanks so much.

JOSEPHGILBY commented 7 months ago

@Bajchos It's a workaround, but don't these steps lead to the source being stored locally rather than in a volume (which could have performance impacts)?

kiweezi commented 7 months ago

@JOSEPHGILBY

I've not had any performance issues myself.

I think this is because the files are cloned and modified inside WSL. So in my context, the volume isn't needed.

Use WSL vscode extension and command (CTRL+SHIFT+P) WSL: Connect to WSL.

Outside of that, there may be a performance benefit to having your files in a container volume, rather than local to WSL. But as of now, I'm yet to feel the difference.

thorgrimjansrud commented 6 months ago

Seems from the docker documentation that we normally want to use volumes instead of directory mounts; "volumes are the best way to persist data in Docker." and "bind mounts have limited functionality compared to volumes". I guess "open folder in container" that do work uses directory mount. It may be a solution here, I did not test yet. It's such an basic thing to use docker volumes so this option should be fixed for sure..

Something like:

docker volume create <your-volume>

devcontainer.json:

"workspaceMount": "source=<your-volume>,target=/workspace,type=volume",
"workspaceFolder": "/workspace"

We can then clone repo inside workspace or the volume-path..

rgalonso commented 3 months ago

I was having a similar issue with HTTPS cloning, even though it was working fine for me using SSH. I determined that this was due to the ls-remote silently failing while asking for credentials. (See my comment to a similar issue here.)

The reason it was working for me with SSH was because I had an agent running in WSL with my key already loaded. As soon as I removed my key, I was experiencing exactly the issue described here.

So with all of that in mind, I'd recommend the following. Independent of VSCode, are you able to open a WSL terminal and run the git ls-remote using the SSH URL? If you're prompted for credentials, then this means that WSL is either not running an SSH agent or your key isn't loaded in there.

The easiest thing to do is run an independent SSH agent in WSL and add the key to it. That should allow VSCode to operate as you'd expect. It is possible to have WSL access your Windows-side SSH agent. I'm doing that, but admittedly it's a bit more advanced so most users may not want to deal with it. Let me know if you'd like me to share those details. (But I may be slow in responding!)

Malix-Labs commented 1 month ago

I got the same problem and it's still happening since a very, very long time

It is currently impossible to make VSCode clone a private git repository in a container volume on Linux, unless proven otherwise

It apparently is an issue with authentication, because when I do try again, it asks me for my username and password

This is a big issue, can this be escalated @chrmarti ?

Configuration

Malix-Labs commented 1 month ago

Workaround

Windows

This workaround is better than https://github.com/microsoft/vscode-remote-release/issues/9163#issuecomment-2024833120 : cloning the git repository in the container volume instead of in WSL (which is optimal)

  1. Remote Repositories: Open Remote Repositories...
  2. Select a private repository you have access to
  3. Once it is opened : Dev Containers: Continue Working in Container Volume

Linux

Adaptation from https://github.com/microsoft/vscode-remote-release/issues/9163#issuecomment-2024833120

Important Precision : cloning the git repository locally instead of the container volume (which is suboptimal) On Linux VSCode, there is currently no known way to clone a git repository in a container volume instead of locally (see https://github.com/microsoft/vscode-remote-release/issues/9163#issuecomment-2438406744) if you found one, please mention me so I can update this comment

  1. Git: Clone (clone locally)
  2. Dev Containers: Reopen in Container
chrmarti commented 1 month ago

I'll connect the terminal's input to the git clone command. A better approach would be to use GIT_ASKPASS and show that in VS Code's input box UI, but that is more involved.

Malix-Labs commented 1 month ago

@chrmarti has this been solved?

As in "it works in VSCode out of the box" ?

If not, please reopen this issue

For now, it seems that it is impossible to make VSCode clone private repositories in container volume (pleas read https://github.com/microsoft/vscode-remote-release/issues/9163#issuecomment-2438570689)

Here are the reproduction steps:

  1. Configure github for public/private key access
  2. Configure the key with a passphrase
  3. In VS Code, Open command palette
  4. enter: Dev Container: Clone Repository in Container Volume
  5. Observe no response
chrmarti commented 1 month ago

This is now available in Dev Containers 0.390.0-pre-release. You can click the link in the progress notification in the lower right of the window to reveal the log terminal and that now forward keyboard input to git clone while that is running. Please give that a try and let me know if that makes it work for you. Thanks.

Malix-Labs commented 1 month ago

Just tested, it is not working

Image

If this is a new issue and there is no known duplicate, please let me know so I can create it

Reproduction steps

  1. >Dev Containers: Clone Repository in Container Volume...
  2. Select a private repository you have access to

Further Read

https://github.com/microsoft/vscode-remote-release/issues/9163#issuecomment-2438406744

Logs

VSCode

Local Host

                       ▅▃▂▄▅▃▄            malix@malix-pc
                ▂▃▃▄▅▆▆▇▗▆▅▄▃▂▝▝    
               ┏▅▆▆┈▅▇▇▁▚▅▆▍▇▆▇▝▖       󱋩  bluefin-dx-nvidia:stable 🔐
               ▅╸▄▃━━▆▁▗▘▝▘▇━╻▗┽▝▖      󰣛  Bluefin-dx 40 (FROM Fedora Silverblue)
                ▅▄▄▖▆━▃╸▖╶▄▂▄▖╲▆▃▖▖       Linux 6.11.3-200.fc40.x86_64
            ▁▃▂     ▝▖┏▗▄┊▅▄▇▘╹▘┒▉▋     󰅐  11 hours, 29 mins
      ▂▃▂▂▃▄▄▆▆▇▅▄▄▂▂▃▘▖▎╲┑▃▎╶▖▁▝▎╻     󰔠  Forged on May 19 2024
▄▆▂▄▆▇▇▂▇┮▆▁▃▂▂─▃▄┰┈▆▃╺▖▝▊▝▃▘▎┛┫▍▋▍ 
▇━━╸▋╺╁▝▅▄▄▄▂╼▇▂▃▄▅▆▘━▏▂┭▃▝▖▅▚▍╴▄▋▍     󰾰  82JU (LEGION 5 15ACH6H)
▄▆╼┰┅▃▘╻▁▃━╼▖┅▃┈━▅▅┘▁╹▇▎╳━▎▚╸╵▅▃▗▋      󰻠  AMD Ryzen 7 5800H (16) @ 4.46 GHz
╸┎╌▆▆┗╺▘▘▖╸▇┗▖▅▂▎┷▂━╴┒╴┞╏╵▖▚▃╴┍▚▗       󰍛  NVIDIA GeForce RTX 3060 Mobile / Max-Q
┇▗┉╀┴╌▚▗▏┘╶┴▉▖▖╺▅┏▄▂╴┈┅▗▗━▏▝╱╱▘▘        󰍛  AMD Radeon Vega Series / Radeon Vega Mobile Series [Integrated]
▁▊▍▍╶┊╱▝┅╶╼┲┕┨▌▅╱╴▗▘└▏╄▝▗▆▘▂▚▄          󰧑  6.77 GiB / 13.49 GiB (50%)
╸┛▖╿▍╎▊┼┏┉╇▇┓▗▊▋┢╶┱▍▘┯╴┈▚╲▃▆▗▃▃▁          61.57 GiB / 952.60 GiB (6%) - btrfs [Read-only]
▁▁┠▖▍▃▃▃▆▅▄▄▘▄▊┨╱╸╵▗╺▃┚┛┈┫▝╲▗▃┣▁╸       󰍹  2560x1440 @ 144 Hz (as 2048x1152) in 27" [External] *
▃▃▃╱▘▃┒▗┦▅▗▚╋▂▂▝╺╱▆▂╼┻┏▄┗▖┈▊▖▉┊▋        󰍹  1920x1080 @ 165 Hz (as 1536x864) in 15" [Built-in]
  ╵▅▃▃▃┎▗▘▉▌╺╹╸▗▘▅▅┍┛╱▖┃▄▄▗▋▌▋╍▏          100% [AC Connected]
  ▉▂▘▘▃▄▖▖▆▅┈┍▄▃▄▄▄▗▍▅▖▄▘▎▚╲▏▌┺     
  ▍▘▂▗▗  ▎▆▖▎▆▘    ▝▌▝╱▃▘▄▍▗▅           󰕮  GNOME 46.5
   ▖▁╏▌  ▘▄▝▎ ▅▅━┑▖ ▄▄▅▂▘▉▃               Mutter (Wayland)
   ▝▌╆▎ ╴▗▆▇▅╍▄▌  ▘                       fish 3.7.0
    ▍▏▝▁▆▌┃▏▁▆┆▋▅▇▅                       Ptyxis 46.7
    ▋▌▅┛┰▇━▁━▅▅▅▅▅▆                     󰏖  2294 (rpm), 62 (flatpak), 58 (brew)
     ▄▃▃▂▃▂▘▅▘
chrmarti commented 4 weeks ago

@Malix-Labs Your screenshot shows a "Username" text input box at the top of the window. Does that not work?

Malix-Labs commented 4 weeks ago

@chrmarti no, username and passwords login has been deprecated in 2021 from the github api

chrmarti commented 3 weeks ago

It might expect a token as the password. Is that not what you expect? What does it ask you for when you git clone the repository in a local terminal?

Malix-Labs commented 3 weeks ago

@chrmarti

It might expect a token as the password

Yes, this is the whole point of that opened issue

Is that not what you expect?

You noticed that the VSCode GUI asks for a password, not a passphrase Inputting the passphrase in the password input doesn't work either

Could you please try to reproduce it on a Linux machine ? It is a simple 2 steps process : https://github.com/microsoft/vscode-remote-release/issues/9163#issuecomment-2452708405

dgsmith1212 commented 3 days ago

Workaround - Clone Private Repo in Container Volume

Provides a workaround based on Git Personal Access Tokens.

I know the OP said "key pair with passphrase" but because the vscode clone logic is using the HTTPS protocol, the correct approach for system to system authn/authz is to use a webauth token.

Configuration

Based on comments in this issue thread, I took the pre-release version of DevContainers. With this vanilla configuration, attempting to clone a private repo into a container volume still hangs on this command:

git ls-remote https://github.com/<YourGitID>/<YourGitRepo>.git

I am not certain but it seems the clone repo to volume logic is trying to launch a config/setup container that doesn't have access to the necessary git authn/authz credentials. This workaround gives it and other containers the necessary credential access.

Here's what I did to resolve. Note this is with HTTPS and GIT acess tokens because the clone repo logic is using HTTPS.

  1. at github.com, configure a fine grained personal access token with broad read/write access

    • click your id_icon (upper right) -> settings -> developer settings
    • pick personal access tokens -> fine grained tokens
    • authenticate to github
    • fill out the form from top to bottom
    • set permission to read/write for many (all?) of the repo permission attributes
    • generate and save the token in a safe place

  2. from Windows, open a WSL prompt (powershell->wsl)

    • navigate to your home directory cd ~

  3. at WSL prompt, configure git to use a globally available credential helper with a persistent credential store

    git config --global credential.helper store

  4. copy your git authn/authz token to the clipboard

  5. back at the WSL prompt, re-enter the failing command from above:

    git ls-remote https://github.com/<YourGitID>/<YourGitRepo>.git

  6. the git credential helper will prompt for userID and password (authentication token)

    • enter your github userID at the ID prompt
    • paste your access token at the Password prompt
    • a .git-credentials file will be created in your WSL home folder.
    • the .git-credentials file is plaintext and contains sensitive information. Given your situation, you should ensure it has adequate protection and adjust this procedure as required.
    • the file could be hand generated instead of entering the failed command and responding to prompts (not tested)

  7. the clone repo in container volume command should now work and the credential helper with its store should be persistent and available to all containers.

Hope this works and helps.

Malix-Labs commented 2 days ago

No idea if this would work on Linux, but thanks for sharing this workaround

Would be nice if it wouldn't required though, looking forward for a fix :)

dgsmith1212 commented 2 days ago

No idea if this would work on Linux, but thanks for sharing this workaround

Would be nice if it wouldn't required though, looking forward for a fix :)

I don't think there's anything in the workaround that is windows specific or dependent. Give it a try it by replacing all steps that are "do at WSL prompt" with "do at Linux prompt."

chrmarti commented 7 hours ago

@Malix-Labs The screenshot shows you are using https, maybe you used the GitHub repository picker in VS Code, that always gives you https. Try copying the ssh connection string from the GitHub repository page and paste that when asked for the repository.

chrmarti commented 7 hours ago

I just found that you then get stuck later when git clone needs the passphrase. This needs more investigation.

As a workaround you could add your key to the local ssh-agent which gets automatically forwarded for Dev Containers. Run ssh-add <path to private key> locally to do so.