Open eveerman opened 2 years ago
Just as a data point, I think I'm seeing similar behavior. I typically call podman from a Git for Windows / Git Bash environment, with SSH_AUTH_SOCK pointing to the the socket file created by Pageant. When I try most podman commands, I get the "failed to create sshClient" error in the OP here. If I unset SSH_AUTH_SOCK
first, all commands work fine for that session.
For me, Event Viewer shows events for each failed attempt: "ssh-agent: error: cannot retrieve client impersonation token" and "ssh-agent: fatal: failed to retrieve client details". Maybe Podman's ssh client is connecting to the named pipe, but Windows ssh-agent can't associate a user account for whatever reason.
This behavior prevents using devcontainers in VSCode without WSL, because the ms-vscode-remote.remote-containers
extension sets SSH_AUTH_SOCK to a default if unset. I was able to get Windows ssh-agent, Podman, and VSCode to cooperate in a terminal session by forwarding the pipe as a socket with rupor-github/wsl-ssh-agent, setting SSH_AUTH_SOCK
to match, and starting code
. Be warned that having the variable set breaks ssh-add
and probably other things too.
I wanted to confirm here that https://github.com/containers/podman/pull/15094 does not fix this issue. I just updated Podman to 4.2.0, and I still see the same failure until I unset SSH_AUTH_SOCK
.
Can you open a fresh issue? Probably easier than reopening this, and there isn't much discussion here.
This issue is still open, I just wanted to proactively make sure nobody came by to say "maybe this is fixed by #15094".
A friendly reminder that this issue had no activity for 30 days.
Is there a workaround for this by setting a windows SSH_AUTH_SOCK environment variable? If so what would someone need to set it to in order to get devcontainers in VSCode without WSL working?
Just to clarify, do you mean is there a value you can use for SSH_AUTH_SOCK
that "fixes" podman, rather than just unsetting the variable (as I mentioned upthread)? Unsetting does allow the command to run successfully.
Can you help me understand where to run that command? I'm on windows and unset isn't a command.
Ah right, sorry, you did say "without WSL" -- I use Git Bash (bash
distributed with Git for Windows) but the effect is much the same. In Windows, I believe to "remove" an environment variable, you use set SSH_AUTH_SOCK=
(with nothing after the equal-sign).
No luck. I am thinking that unsetting SSH_AUTH_SOCK isn't a fix for me. I'm on windows using cmder instead of gitbash. if I do a printenv I don't see a SSH_AUTH_SOCK variable.
this is the error I am seeing in vscode:
[11674 ms] Remote-Containers 0.251.0 in VS Code 1.71.2 (74b1f979648cc44d385a2286793c226e611f59e7).
[11674 ms] Start: Run: podman version --format {{.Server.APIVersion}}
[11750 ms] Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: dial unix \\.\pipe\openssh-ssh-agent: connect: No connection could be made because the target machine actively refused it.
This is what I see on the command line:
λ podman system connection list
Name URI Identity Default
podman-machine-default ssh://user@localhost:65061/run/user/1000/podman/podman.sock C:\Users\jcubi\.ssh\podman-machine-default false
podman-machine-default-root ssh://root@localhost:65061/run/podman/podman.sock C:\Users\jcubi\.ssh\podman-machine-default true
I am wondering what ssh client vscode is using. I don't have an ssh client installed system wide. I'm just using the one in cmder and I don't have that one on the windows path for other terminals to use.
Turns out that I do have ssh installed at C:\Windows\System32\OpenSSH\ssh.exe, but, I do not see an SSH_AUTH_SOCK variable as being set when looking through the child items for :env in powershell
Podman machine relies on physical host key auth and manages those keys for you. It sounds like from your reproducer you are deleting the keys that podman machine requires. Is that right? Or are you saying that simply having an agent and SSH_AUTH_SOCK set by that agent breaks the local key auth? Podman commands should fall back to the key directly, but if that’s not happening then I agree we need to look at that.
If all of you that are affected by this could provide some more detail about your use cases (e.g using remote but not machine) that would be a big help. Thanks!
I have not been monitoring this so much. SO the issue as I recall did very much seem that any setting of SSH_AUTH_SOCK on windows cause issue. For one as having the variable set meant that for some reason the local key auth then did not work. both for remote and for machine, though I can't state for certain.
regardless as of 4.5.0 (at least I was a bit lax with updates) it works exactly as expected:
~PS C:\Users\qdo0obp> ssh-add -L ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMRA11BL+vy8kUlnQ310uyjzBRyhUSG6TOHPseJSsv3/ cardno:14 702 056 PS C:\Users\qdo0obp> podman ps --log-level debug time="2023-04-16T10:26:15+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level debug" time="2023-04-16T10:26:15+02:00" level=debug msg="Called ps.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe ps --log-level debug)" time="2023-04-16T10:26:15+02:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\qdo0obp\\\\.ssh\\\\podman-machine-default\" SHA256:rkhoxLoQKvccf47/PRvGF5GV1k7PhQ60w7rhdse7B4Q ssh-ed25519" time="2023-04-16T10:26:15+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.5.0/libpod/_ping" time="2023-04-16T10:26:15+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.5.0/libpod/containers/json" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES time="2023-04-16T10:26:15+02:00" level=debug msg="Called ps.PersistentPostRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe ps --log-level debug)" time="2023-04-16T10:26:15+02:00" level=debug msg="Shutting down engines" PS C:\Users\qdo0obp> podman pull almalinux Resolved "almalinux" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull docker.io/library/almalinux:latest... Getting image source signatures Copying blob sha256:63c7cbfce3f306a83bb69ed11284ee1fdcd3fdbeec4769e30e03123cd7ced99d Copying config sha256:f4b24db3d26dc26d30bec138133d0c820bed20d98bbd2dc91b7386d0df2cee35 Writing manifest to image destination Storing signatures f4b24db3d26dc26d30bec138133d0c820bed20d98bbd2dc91b7386d0df2cee35 PS C:\Users\qdo0obp> podman images --log-level debug time="2023-04-16T10:27:00+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level debug" time="2023-04-16T10:27:00+02:00" level=debug msg="Called images.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe images --log-level debug)" time="2023-04-16T10:27:00+02:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\qdo0obp\\\\.ssh\\\\podman-machine-default\" SHA256:rkhoxLoQKvccf47/PRvGF5GV1k7PhQ60w7rhdse7B4Q ssh-ed25519" time="2023-04-16T10:27:00+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.5.0/libpod/_ping" time="2023-04-16T10:27:00+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.5.0/libpod/images/json" REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/almalinux latest f4b24db3d26d 4 days ago 196 MB time="2023-04-16T10:27:00+02:00" level=debug msg="Called images.PersistentPostRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe images --log-level debug)" time="2023-04-16T10:27:00+02:00" level=debug msg="Shutting down engines" PS C:\Users\qdo0obp> podman run -d --rm --name testy f4b24db3d26d /bin/bash -c "sleep 3600" fe229400d5c64a16531d8cc98ee0134c787e15938f2efdd365ca1ad42566d8cb PS C:\Users\qdo0obp> podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fe229400d5c6 docker.io/library/almalinux:latest /bin/bash -c slee... 2 seconds ago Up 2 seconds testy PS C:\Users\qdo0obp> podman ps --log-level debug time="2023-04-16T10:27:28+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level debug" time="2023-04-16T10:27:28+02:00" level=debug msg="Called ps.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe ps --log-level debug)" time="2023-04-16T10:27:28+02:00" level=debug msg="SSH Ident Key \"C:\\\\Users\\\\qdo0obp\\\\.ssh\\\\podman-machine-default\" SHA256:rkhoxLoQKvccf47/PRvGF5GV1k7PhQ60w7rhdse7B4Q ssh-ed25519" time="2023-04-16T10:27:28+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.5.0/libpod/_ping" time="2023-04-16T10:27:28+02:00" level=debug msg="DoRequest Method: GET URI: http://d/v4.5.0/libpod/containers/json" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fe229400d5c6 docker.io/library/almalinux:latest /bin/bash -c slee... 8 seconds ago Up 8 seconds testy time="2023-04-16T10:27:28+02:00" level=debug msg="Called ps.PersistentPostRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe ps --log-level debug)" time="2023-04-16T10:27:28+02:00" level=debug msg="Shutting down engines" PS C:\Users\qdo0obp>
~
just re-read that and sadly still does not work as hoped. - looking closer that is not MY key being used but the keyfile create by podman machine init.
So on the plus side: having SSH_AUTH_SOCK set no longer seems to break it entirely BUT other than podman machine ssh
none of the commands work with whatever key should be behind the auth sock.
for example: add a new connection to the podman machine with no ident file:
PS C:\Users\qdo0obp> podman system connection add wut user@localhost:30356/run/user/1000/podman/podman.sock
PS C:\Users\qdo0obp> podman system connection list
Name URI Identity Default
coreos ssh://core@192.168.157.10:22/run/user/1000/podman/podman.sock C:\Users\qdo0obp\Documents\KEYS\id_fedoreCoreOS false
podman-machine-default ssh://user@localhost:30356/run/user/1000/podman/podman.sock C:\Users\qdo0obp\.ssh\podman-machine-default true
podman-machine-default-root ssh://root@localhost:30356/run/podman/podman.sock C:\Users\qdo0obp\.ssh\podman-machine-default false
wut ssh://user@localhost:30356/run/user/1000/podman/podman.sock false
see my ssh key via SSH_AUTH_SOCK:
PS C:\Users\qdo0obp> ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMRA11BL+vy8kUlnQ310uyjzBRyhUSG6TOHPseJSsv3/ cardno:14 702 056
local key file rename just to hide it:
PS C:\Users\qdo0obp> gci .\.ssh\podman-machine-default*
Directory: C:\Users\qdo0obp\.ssh
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 17/04/2023 12:41 94 podman-machine-default.pub
-a--- 17/04/2023 12:41 399 podman-machine-defaultNOPE
Podman machine ssh works as expected:
PS C:\Users\qdo0obp> podman -c wut machine ssh --log-level debug
time="2023-04-17T13:06:39+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level debug"
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
time="2023-04-17T13:06:39+02:00" level=debug msg="Executing: ssh [-i C:\\Users\\qdo0obp\\.ssh\\podman-machine-default -p 30356 user@localhost -o UserKnownHostsFile /dev/null -o StrictHostKeyChecking no]\n"
Warning: Identity file C:\Users\qdo0obp\.ssh\podman-machine-default not accessible: No such file or directory.
Warning: Permanently added '[localhost]:30356' (ED25519) to the list of known hosts.
Last login: Mon Apr 17 13:04:00 2023 from ::1
[user@xps9500 ~]$ cat .ssh/authorized_keys
#ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOmUIohoIyk7jQAIG+P8iTZkQq7dLvePrNunUOSnCEBE root@xps9500
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMRA11BL+vy8kUlnQ310uyjzBRyhUSG6TOHPseJSsv3/ cardno:14 702 056
but then other direct commands fail:
PS C:\Users\qdo0obp> podman -c wut ps --log-level debug
time="2023-04-17T13:07:28+02:00" level=info msg="C:\\Program Files\\RedHat\\Podman\\podman.exe filtering at log level debug"
time="2023-04-17T13:07:28+02:00" level=debug msg="Called ps.PersistentPreRunE(C:\\Program Files\\RedHat\\Podman\\podman.exe -c wut ps --log-level debug)"
time="2023-04-17T13:07:28+02:00" level=debug msg="Found SSH_AUTH_SOCK \"\\\\\\\\.\\\\pipe\\\\ssh-pageant\", ssh-agent signer enabled"
Error: dial unix \\.\pipe\ssh-pageant: connect: No connection could be made because the target machine actively refused it.
time="2023-04-17T13:07:28+02:00" level=debug msg="Shutting down engines"
@n1hility it certainly does seem to rely on physical identity key auth for the majority of its functionality but not for an interactive ssh login?
It looks like it uses it's own ssh driver for actual podman commands but then when calling ssh
it instead is happy to use default system ssh client?
as a note on Linux (alma8) it seems to work more as expected with the standard SSH_AUTH_SOCK being read and used:
[erik.veerman@vm03-ams ~]$ podman -c coreos ps --log-level debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman -c coreos ps --log-level debug)
DEBU[0000] Found SSH_AUTH_SOCK "/tmp/ssh-H5CF5jgJGh/agent.493768", ssh-agent signer enabled
DEBU[0000] SSH Agent Key SHA256:q4EoCC2l7dn6UD579Q4xZyWHigxEcghi0i9l0+I4guY ssh-rsa
DEBU[0000] SSH Agent Key SHA256:1dkrBgfFrDcN2e1601AFLrhP8tC3+nJ+N19uNNfxs/Q ssh-rsa
DEBU[0000] SSH Agent Key SHA256:5rk4fyyGn/RWHlvX6viKskm6JunH1sXdqqH22v/XBoU ssh-rsa
DEBU[0001] DoRequest Method: GET URI: http://d/v4.3.1/libpod/_ping
DEBU[0001] Found SSH_AUTH_SOCK "/tmp/ssh-H5CF5jgJGh/agent.493768", ssh-agent signer enabled
DEBU[0002] SSH Agent Key SHA256:1dkrBgfFrDcN2e1601AFLrhP8tC3+nJ+N19uNNfxs/Q ssh-rsa
DEBU[0002] SSH Agent Key SHA256:q4EoCC2l7dn6UD579Q4xZyWHigxEcghi0i9l0+I4guY ssh-rsa
DEBU[0002] SSH Agent Key SHA256:5rk4fyyGn/RWHlvX6viKskm6JunH1sXdqqH22v/XBoU ssh-rsa
DEBU[0003] DoRequest Method: GET URI: http://d/v4.3.1/libpod/_ping
DEBU[0003] DoRequest Method: GET URI: http://d/v4.3.1/libpod/containers/json
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
DEBU[0003] Called ps.PersistentPostRunE(podman -c coreos ps --log-level debug)
Makes me think that as a workaround I could use WSL with my auth passed through and then utilise podman remote from there. Bit circular though.
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Having SSH_AUTH_SOCK set on windows seems to break many of the windows client functions and/or podman windows client simply does not use the ssh-agent? This happens when I am I am using a yubikey and an agent forwarding program OR just windows openssh's agent & keys.
podman machine ssh
works in both cases above without issue but, e.g.podman ps
does not. Tried to use putty/plink, tried blanking the "identity" field in the connection, tried setting identity as PLINK or the yubikey pipe but neither was happySteps to reproduce the issue:
use generated key pair to copy pub key(s) to "machine":
...
podman machine ssh
...rename/remove private key
do .. something eg
podman ps
Describe the results you received:
Slightly different if using SSH_AUTH_SOCK and a yubikey:
Describe the results you expected: see some containers :(
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version
:Output of
podman info --debug
:Package info (e.g. output of
rpm -q podman
orapt list podman
):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
Physical, windows :(