sshpass does not work with Windows Terminal app and msys2 console

brendan-simon-indt commented 1 year ago

Description / Steps to reproduce the issue

sshpass works with ssh if I run from a stock standard msys2 terminal (or mingw64, clang, ucrt, ...).

sshpass works with Windows Terminal app, if using sshpass with windows ssh, but not with msys2 ssh.

sshpass does not work with Windows Terminal app with a msys2, mingw64, ... terminal in a tab. I get absolutely no output. I don't know if this is an issue with sshpass or Windows Terminal app.

NOTE: the ssh command after the sspass does work if given by itself (I have ssh rsa keys setup)

Install Windows Terminal app from the Windows Store https://apps.microsoft.com/store/detail/windows-terminal/9N0DX20HK701?hl=en-au&gl=au&rtc=1
Open up an msys terminal from the drop down menu.
Issue sspass command. e.g. sshpass -puserpassword user@host ls

Expected behavior

Output to be displayed in terminal.

Actual behavior

absolutely no output

Verification

[X] I have verified that my MSYS2 is up-to-date before submitting the report (see https://www.msys2.org/docs/updating/)

Windows Version

MSYS_NT-10.0-19044

Are you willing to submit a PR?

unlikely

sskras commented 1 year ago

Where did you get / how have you built the sshpass binary?

brendan-simon-indt commented 1 year ago

Where did you get / how have you built the sshpass binary?

It's an msys2 package.

$ pacman -Ss sshpass
msys/sshpass 1.10-1 (net-utils) [installed]
    Fool ssh into accepting an interactive password non-interactively

pinyht commented 1 year ago

same problem

Biswa96 commented 1 year ago

sshpass works with ssh if I run from a stock standard msys2 terminal

If sshpass works with msys2's default terminal program (mintty), why is this not an issue of Windows Terminal?

sskras commented 1 year ago

@brendan-simon-indt commented 3 weeks ago:

Open up an msys terminal from the drop down menu.

Issue sspass command. e.g. sshpass -puserpassword user@host ls

@brendan-simon-indt, @pinyht:

What do you get if you run where ssh.exe after the last step (in the same environment)? Maybe your combo picks up the wrong ssh binary (the Windows-native one).

brendan-simon-indt commented 1 year ago

If I run from an msys2 terminal/shell (Windows launcher icon from start menu), it works with direct ssh and manual password entry and with sshpass. Here is the output.

$ ssh brendansimon@192.168.56.101 "uname -a"
brendansimon@192.168.56.101's password:
Linux osboxes 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 GNU/Linux

# with wrong password
$ sshpass -p "BAD" ssh brendansimon@192.168.56.101 "uname -a"
Permission denied, please try again.

# with correct password
$ sshpass -p "GOOD" ssh brendansimon@192.168.56.101 "uname -a"
Linux osboxes 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 GNU/Linux

$ where ssh
C:\msys64\usr\bin\ssh.exe

Doing the same things from Windows Terminal app with msys2 shell, works using ssh and manual password, but I get no output if running via sshpass.

$ ssh brendansimon@192.168.56.101 "uname -a"
brendansimon@192.168.56.101's password:
Linux osboxes 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 GNU/Linux

# wrong password - there is a noticeable delay before prompt returns (no output)
$ sshpass -p "XXX" ssh brendansimon@192.168.56.101 "uname -a"

# correct password - prompt returns quickly (no output)
$ sshpass -p "GOOD" ssh brendansimon@192.168.56.101 "uname -a"

$ where ssh
C:\msys64\usr\bin\ssh.exe

It could be a Windows Terminal app issue, but why does ssh work ok, but not with sshpass?

sskras commented 1 year ago

I would try running sshpass under strace in both setups. Comparing the outputs might give some insight. Beware that they would be over 1000-2000 lines each.

Toxsch commented 1 year ago

Encountered the same problem, add a little verbose information:

$ sshpass -v -e ssh user@host
SSHPASS: searching for password prompt using match "assword"
user@host's password:
SSHPASS: detected prompt. Sending password.
SSHPASS: read:

pinyht commented 1 year ago

I found sshpass only work with mintty and conemu, and if use conemu, must start with conemu-msys2-64.exe, I can use conemu-msys2-64.exe with bash.exe in windows terminal, sshpass will work, but it can't enable xterm-256color,maybe have some bugs, if direct use msys2_shell.cmd or bash.exe, sshpass can't work, not just windows terminal, i have already tried windows terminal, conemu, cmder, tabby, hyper.js, all can't work

sskras commented 1 year ago

@brendan-simon-indt commented on Feb 18:

Install Windows Terminal app from the Windows Store https://apps.microsoft.com/store/detail/windows-terminal/9N0DX20HK701?hl=en-au&gl=au&rtc=1

Open up an msys terminal from the drop down menu.

I would like to reproduce this. Just installed Windows Terminal version 1.16.10261.0:

But MSYS2 terminal isn't here:

Did you configure it by hand or was it added there automatically on your machine?

mmuetzel commented 1 year ago

@sskras: Instructions for setting up the Windows Terminal for the MSYS2 environments are here: https://www.msys2.org/docs/terminals/

brendan-simon-indt commented 1 year ago

@sskras I can't remember now. This is what is in my settings. I guess I may have set it up myself.

sskras commented 1 year ago

Thanks, it comes probably from here: https://www.msys2.org/docs/terminals/#windows-terminal

// This makes UCRT64 the default shell
"defaultProfile": "{17da3cac-b318-431e-8a3e-7fcdefe6d114}",
"profiles": {
  "list":
  [
    // ...
    {
      "guid": "{17da3cac-b318-431e-8a3e-7fcdefe6d114}",
      "name": "UCRT64 / MSYS2",
      "commandline": "C:/msys64/msys2_shell.cmd -defterm -here -no-start -ucrt64",
      "startingDirectory": "C:/msys64/home/%USERNAME%",
      "icon": "C:/msys64/ucrt64.ico",
      "font": 
      {
        "face": "Lucida Console",
        "size": 9
      }
    },
    {
      "guid": "{71160544-14d8-4194-af25-d05feeac7233}",
      "name": "MSYS / MSYS2",
      "commandline": "C:/msys64/msys2_shell.cmd -defterm -here -no-start -msys",
      "startingDirectory": "C:/msys64/home/%USERNAME%",
      "icon": "C:/msys64/msys2.ico",
      "font": 
      {
        "face": "Lucida Console",
        "size": 9
      }
    },
    // ...
  ]
}

sskras commented 1 year ago

I can confirm that the issue is present in MSYS2. The tested WT profile:

            {
                "commandline": "C:/msys64/msys2_shell.cmd -defterm -here -no-start -msys",
                "font": 
                {
                    "face": "Lucida Console",
                    "size": 9.0
                },
                "guid": "{71160544-14d8-4194-af25-d05feeac7233}",
                "icon": "C:/msys64/msys2.ico",
                "name": "MSYS / MSYS2",
                "startingDirectory": "C:/msys64/home/%USERNAME%"
            },

This can also be reproduced without Windows Terminal by simply running MSYS2 from Start > Run as follows:

C:/msys64/msys2_shell.cmd -defterm -here -no-start -msys
C:/msys64/msys2_shell.cmd -defterm
C:/msys64/usr/bin/bash.exe --login -i

Without the -defterm the script opens the standalone MinTTY window where the I/O works fine.

This also affects Cygwin. The tested WT profile:

            {
                "commandline": "C:/cygwin64/bin/bash.exe --login -i",
                "guid": "{cdaa0fd5-e104-4561-9b7c-23177717bc48}",
                "hidden": false,
                "icon": "C:/cygwin64/Cygwin-Terminal.ico",
                "name": "Cygwin",
                "startingDirectory": "C:/cygwin64/home/%USERNAME%"
            }

Tested command lines:

C:/cygwin64/Cygwin.bat
C:/cygwin64/bin/bash.exe --login -i

sskras commented 1 year ago

I tested printing version of the SSH client while running under sshpass:

SSHPASS=. sshpass -e ssh -V

If I run this from MinTTY, version is shown:

saukrs@DESKTOP-O7JE7JE MSYS ~
$ SSHPASS=. sshpass -e ssh -V
OpenSSH_9.3p1, OpenSSL 3.1.0 14 Mar 2023

saukrs@DESKTOP-O7JE7JE MSYS ~
$

Now if run this from console I/O based environment (cmd.exe, Windows Terminal), the stdout is lost:

saukrs@DESKTOP-O7JE7JE MSYS ~
$ SSHPASS=. sshpass -e ssh -V

saukrs@DESKTOP-O7JE7JE MSYS ~
$

Not sure if it counts as a reduced test case.

But clearly all this is related to the initial terminal I/O setup.

sskras commented 1 year ago

I tried stracing the sshpass in both cases, unifying (normalizing) some identifactors/addresses in the content, and then compared both. I noticed that:

set of env-vars are quite different
in the terminal (MinTTY) there is MSYSCON=mintty.exe set, \ while in console (cmd.exe) is missing such var.
the terminal gets MAKE_TERMOUT=/dev/pty0, \ the console gets MAKE_TERMOUT=/dev/cons0.
in the terminal the allocated device is /dev/pty1, \ in the console the allocated device is /dev/pty0. \ \ (Not sure if relevant due to the previous difference) \ \ The minus/red lines mark the console specifics, the plus/green lines mark the terminal specifics:
```
[ptymf] sshpass CPID1 SetThreadName: SetThreadDescription() failed. 00000000 10000000
```
[main] sshpass CPID1 fhandler_pty_master::setup: this 0x800009A90, pty0 opened - from_pty <0x1A4,0x1C0>, to_pty 0x1A0
[main] sshpass CPID1 fhandler_pty_master::setup: this 0x800009A90, pty1 opened - from_pty <0x208,0x210>, to_pty 0x204 [main] sshpass CPID1 fhandler_base::open_with_arch: line 451: /dev/ptmx<0x800009E00> usecount + 1 = 1 [main] sshpass CPID1 fhandler_base::set_flags: flags 0x18002, supplied_bin 0x0 [main] sshpass CPID1 fhandler_base::set_flags: O_TEXT/O_BINARY set in flags 0x10000 [main] sshpass CPID1 fhandler_base::set_flags: filemode set to binary
[main] sshpass CPID1 fhandler_pty_master::open_setup: /dev/ptmx opened pty master for pty0, usecount 1
[main] sshpass CPID1 fhandler_pty_master::open_setup: /dev/ptmx opened pty master for pty1, usecount 1 [main] sshpass CPID1 open: 3 = open(/dev/ptmx, 0x8002) [main] sshpass CPID1 fcntl: fcntl(3, 4, ...) [main] sshpass CPID1 fhandler_base::set_flags: flags 0x1C002, supplied_bin 0x0 [main] sshpass CPID1 fhandler_base::set_flags: O_TEXT/O_BINARY set in flags 0x10000 [main] sshpass CPID1 fhandler_base::set_flags: filemode set to binary [main] sshpass CPID1 fcntl: 0 = fcntl(3, 4, 0x4000) [main] sshpass CPID1 open: open(/dev/tty, 0x0)

in the terminal there are additional dlls loaded (for MinTTY drawing / input operations I guess). Here:

WPID means Windows PID

CPID means Cygwin PID

+--- Process WPID1 (pid: CPID1) loaded C:\Windows\System32\user32.dll at 00007ff887650000
+--- Process WPID1 (pid: CPID1) loaded C:\Windows\System32\win32u.dll at 00007ff886c60000
+--- Process WPID1 (pid: CPID1) loaded C:\Windows\System32\gdi32.dll at 00007ff8894c0000
+--- Process WPID1 (pid: CPID1) loaded C:\Windows\System32\gdi32full.dll at 00007ff886d90000
+--- Process WPID1 (pid: CPID1) loaded C:\Windows\System32\msvcp_win.dll at 00007ff887360000
+--- Process WPID1 (pid: CPID1) loaded C:\Windows\System32\imm32.dll at 00007ff888820000

just before that (in the console) main thread of sshpass does something that exposes a new Cygwin PID (normalized as CPID2 by me). \ \ As I see no fork() or other syscall around, I guess this comes from the MSYS2/Cygwin core/engine. \ No new Cygwin PID is generated in the case of terminal:


[main] sshpass CPID1 build_argv: cmd = 'sshpass -e ssh myhostname id', winshell = 1, glob = 1
[main] sshpass CPID1 build_argv: argv[0] = 'sshpass'
[main] sshpass CPID1 build_argv: argv[1] = '-e'
[main] sshpass CPID1 build_argv: argv[2] = 'ssh'
[main] sshpass CPID1 build_argv: argv[3] = 'myhostname'
[main] sshpass CPID1 build_argv: argv[4] = 'id'
[main] sshpass CPID1 build_argv: argc 5

[main] sshpass CPID1 open_shared: name dd50a72ab4668b33-cons0x150AD8.0, shared 0x1A3000000 (wanted 0x1A3000000), h 0x1AC, m 3, created 0
[main] sshpass CPID1 open_shared: name cygpid.CPID2, shared 0x1A4000000 (wanted 0x1A4000000), h 0x184, m 6, created 0
[main] sshpass CPID1 open_shared: name dd50a72ab4668b33-cons0x5C072C.0, shared 0x1A3000000 (wanted 0x1A3000000), h 0x1C8, m 3, created 1
for the terminal an additional thread called consm gets created in the main process of sshpass. \ It's not present in case of console:
```
+--- Process WPID1 (pid: CPID1) thread WTID4 created
```
[consm] sshpass CPID1 cygthread::stub: thread 'consm', id 0x3EC8, stack_ptr 0x29DCCB0
[consm] sshpass CPID1 SetThreadName: SetThreadDescription() failed. 00000000 10000000

the tty->pgid and tty->sig set notably different:

for console tty sid=CPID2, pgid=CPID7 \ (CPID2 is seen above but origin is unknown) \ (CPID7 is rather strange as this CPID is next seen only far below in the trace)

for terminal tty sid=0, pgid=CPID1 \ (CPID1 is the main PID of sshpass)


[main] sshpass CPID1 fhandler_base::open_with_arch: line 451: /dev/cons0<0x800008D60> usecount + 1 = 1
[main] sshpass CPID1 fhandler_base::set_flags: flags 0x10002, supplied_bin 0x0
[main] sshpass CPID1 fhandler_base::set_flags: O_TEXT/O_BINARY set in flags 0x10000
[main] sshpass CPID1 fhandler_base::set_flags: filemode set to binary
[main] sshpass CPID1 _pinfo::set_ctty: old no ctty, ctty device number 0xFFFFFFFF, tc.ntty device number 0x30000 flags & O_NOCTTY 0x0
[main] sshpass CPID1 _pinfo::set_ctty: cygheap->ctty 0x0, archetype 0x800008D60
[main] sshpass CPID1 _pinfo::set_ctty: ctty was NULL
[main] sshpass CPID1 _pinfo::set_ctty: line 548: /dev/cons0<0x800008D60> usecount + 1 = 2
[main] sshpass CPID1 _pinfo::set_ctty: /dev/cons0 ctty, usecount 2

[main] sshpass CPID1 _pinfo::set_ctty: attaching ctty /dev/cons0 sid CPID1, pid CPID1, pgid CPID1, tty->pgid CPID7, tty->sid CPID2
[main] sshpass CPID1 _pinfo::set_ctty: attaching ctty /dev/cons0 sid CPID1, pid CPID1, pgid CPID1, tty->pgid CPID1, tty->sid 0 [main] sshpass CPID1 _pinfo::set_ctty: cygheap->ctty now 0x800008D60, archetype 0x800008D60
```
---
```

I lack the exact understanding of inner Cygwin/MSYS2 mechanisms. It still looks to me like its' PTY/PTMX emulation logic is to blame given that sshpass works on a variety of unices:

macOS,
FreeBSD,
NetBSD,
illumOS,
Solaris,
AIX.
Except maybe for the HP-UX.

We need more qualified expertise to narrow the bug down.

After that it might get clearer if the issue really belongs to this repo (instead of msys2/msys2-runtime) or IOW, what's the upstream of the issue: SSHPass or Cygwin itself.

sskras commented 1 year ago

Also ping @Shachar, the developer of sshpass.

jeremyd2019 commented 1 year ago

We need more qualified expertise to narrow the bug down.

For Cygwin tty/console, you can't get more qualified expertise than @tyan0. It would probably be interesting to know if this happens as described under upstream Cygwin (I expect it does), and then whether it still happens under a Cygwin 3.5 snapshot.

Shachar commented 1 year ago

There's a chance the problem isn't with MSYS or sshpass (exactly), but with the version of ssh you're using. See https://sourceforge.net/p/sshpass/bugs/27/ on the upstream sshpass package.

In a nutshell, some ssh versions close their stdout and stderr, and redirect it to their own terminal. Since sshpass doesn't copy between the TTY and stdout/err, the output never goes anywhere.

sskras commented 1 year ago

@jeremyd2019 commented 3 hours ago:

For Cygwin tty/console, you can't get more qualified expertise than @tyan0.

Thanks for inviting a skilled person! :)

It would probably be interesting to know if this happens as described under upstream Cygwin (I expect it does),

I commented 2 days ago that Cygwin fails in a similar manner. I think it's quite up to date:

$ uname -a
CYGWIN_NT-10.0-19044 DESKTOP-O7JE7JE 3.4.6-1.x86_64 2023-02-14 13:23 UTC x86_64 Cygwin

and then whether it still happens under a Cygwin 3.5 snapshot.

I would give it a shot on my machine should I know how to bootstrap this: http://mirror.internode.on.net/pub/cygwin/x86_64/release/cygwin/cygwin-3.5.0-0.298.g41fdb869f998.tar.xz

tyan0 commented 1 year ago

I have confirmed that this issue also occurs in cygwin 3.4.6, but not in the latest cygwin 3.5.0 (Test).

This issue seems to be due to a long-standing limitation of console implementation that the console is inaccessible from other terminals that are associated with other tty. Only console (command prompt, windows terminal etc.) has this limitation but pty (mintty, xterm etc.) does not.

A several months ago, I applied a relatively agressive patch (https://github.com/mirror/newlib-cygwin/commit/3721a756b0d8067cf070e2aedcf3ea5e32e47270) to the cygwin master branch so that GNU screen and tmux get working in console. It seems that this patch solves this issue as well.

Therefore, you can expect that sshpass will work also on console with upcoming v3.5.x.

Cygwin v3.5.x will be released within 2023.

brendan-simon-indt commented 1 year ago

Awesome. Can't come soon enough. Thanks for the amazing work 🙇

sskras commented 1 year ago

ping @lazka: should this be moved into msys2/msys2-runtime repo?

Also I would change the title:

-sshpass does not work with Windows Terminal app and msys2 console
+combination of `sshpass` and `ssh` from MSYS2 renders no output in Windows console

Biswa96 commented 1 year ago

should this be moved into msys2/msys2-runtime repo?

It's ok here, I guess. The issue with sshpass which belongs to this MSYS2-packages repository.

sskras commented 1 year ago

@Biswa96: no, the issue is with msys2-3.4.6/winsup/cygwin:

This issue seems to be due to a long-standing limitation of console implementation that the console is inaccessible from other terminals that are associated with other tty. Only console (command prompt, windows terminal etc.) has this limitation but pty (mintty, xterm etc.) does not.

A several months ago, I applied a relatively agressive patch (mirror/newlib-cygwin@3721a75) to the cygwin master branch so that GNU screen and tmux get working in console. It seems that this patch solves this issue as well.

Specifically with the winsup/cygwin/fhandler/console.cc file, eg. the fhandler_console::set_unit () method.

Because of this applications like tmux also fail:

I expect GNU Screen to fail too (but MSYS2 doesn't build it).

RekaDowney commented 7 months ago

@tyan0 @sskras Hey guys, so now we just need to wait for the release of Cygwin 3.5, right? Could you please let me know where I can find the release roadmap for Cygwin? I tried searching for cygwin roadmap, cygwin release notes or cygwin 3.5.0, but I couldn't find any specific information about the release dates of 3.5.

MehdiChinoune commented 7 months ago

but I couldn't find any specific information about the release dates of 3.5.

https://cygwin.com/pipermail/cygwin-announce/2024-January/011483.html

RekaDowney commented 7 months ago

but I couldn't find any specific information about the release dates of 3.5.

https://cygwin.com/pipermail/cygwin-announce/2024-January/011483.html

Okay, I'll patiently wait for a while.

msys2 / MSYS2-packages