Compiled for cygwin - client_converse: receive: unexpected internal error

denniskniep commented 3 years ago

Hello,

first of all thanks for your awesome work!

I am using your released binaries v2.0.0 with gitforwindows with OpenSSH_8.4p1 and it worked like a charm.

Now I am trying to make it running with MobaXTerm (cygwin)

What i have done so far:

cloned repo on tag v2.0.0
./setup-x86.exe -root <path> -q -P gcc-core -P gcc-g++ -P libssl-deve -P automake -P autoconf -P libtool
autoconf --install
./configure
make
Copied src/.libs/winhello.dll

Executed SSH_SK_PROVIDER=<path>/winhello.dll ssh-keygen -vvv -t ecdsa-sk -f ./fido

After a warning (already read in other issues, that this should be no problem): WinHello API Error: Is User available=0, User=0

Then the following error is returned:

client_converse: receive: unexpected internal error
debug3: reap_helper: pid=7640
reap_helper: helper exited abnormally
Key enrollment failed: unexpected internal error

Any ideas what could be the problem?

tavrez commented 3 years ago

Hello, thanks for reporting. Could you provide more log output? I need to see some lines before client_converse line.

denniskniep commented 3 years ago

sure, hope the complete log helps:

SSH_SK_PROVIDER=<path>/winhello.dll ssh-keygen -vvv -t ecdsa-sk -f ./fido

Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
debug3: start_helper: started pid=7640
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/sbin/ssh-sk-helper
debug1: sshsk_enroll: provider "<path>/winhello.dll", device "(null)", application "ssh:", userid "(null)", flags 0x01, challenge len 0
debug1: sshsk_enroll: using random challenge
debug1: sshsk_open: provider <path>/winhello.dll implements version 0x00070000
init_winhello: WARNING! This should not be like this! WinHello API Error: Is user available=0, User=0.
client_converse: receive: unexpected internal error
debug3: reap_helper: pid=7640
reap_helper: helper exited abnormally
Key enrollment failed: unexpected internal error

tavrez commented 3 years ago

Seems like helper is terminated without doing anything, do you have libfido2 and libcbor installed in your cygwin environment? Winhello do not need them but they are dependencies of ssh-sk-helper so they need to be installed correctly.

On Sat, Jan 23, 2021 at 7:22 PM denniskniep notifications@github.com wrote:

sure, hope the complete log helps:

SSH_SK_PROVIDER=/winhello.dll ssh-keygen -vvv -t ecdsa-sk -f ./fido

Generating public/private ecdsa-sk key pair. You may need to touch your authenticator to authorize key generation. debug3: start_helper: started pid=7640 debug3: ssh_msg_send: type 5 debug3: ssh_msg_recv entering debug1: start_helper: starting /usr/sbin/ssh-sk-helper debug1: sshsk_enroll: provider "/winhello.dll", device "(null)", application "ssh:", userid "(null)", flags 0x01, challenge len 0 debug1: sshsk_enroll: using random challenge debug1: sshsk_open: provider /winhello.dll implements version 0x00070000 init_winhello: WARNING! This should not be like this! WinHello API Error: Is user available=0, User=0. client_converse: receive: unexpected internal error debug3: reap_helper: pid=7640 reap_helper: helper exited abnormally Key enrollment failed: unexpected internal error

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tavrez/openssh-sk-winhello/issues/8#issuecomment-766102438, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFM2DPLS3GTJS2P6XLDDRLS3LWCXANCNFSM4WPXHCMQ .

denniskniep commented 3 years ago

Yes, they are both installed:

tavrez commented 3 years ago

I think it is because of mobaXterm launching terminal as a child process, but I'm not sure Can you add some debug print like this(change msg number each time):

skdebug(__func__, "debug msg 1");

in src/winhello.c before line 296, after line 296 and after line 298 and re-compile and re-run it to see how far my code is going before crash?

denniskniep commented 3 years ago

Added statements here:

        skdebug(__func__, "debug msg 1");

        HWND hWnd = GetForegroundWindow();

        skdebug(__func__, "debug msg 1");

        HRESULT hr = webAuthNAuthenticatorMakeCredential(hWnd, &rpInfo, &userInfo, &WebAuthNCredentialParameters, &WebAuthNClientData, &WebAuthNCredentialOptions, &pWebAuthNCredentialAttestation);

Debug statements not showing up. I added further debug lines to make sure modifications are compiled

sk_enroll: START
init_winhello: TEST v1
init_winhello: WARNING! This should not be like this! WinHello API Error: Is user available=0, User=0.
client_converse: receive: unexpected internal error
debug3: reap_helper: pid=13470
reap_helper: helper exited abnormally
Key enrollment failed: unexpected internal error

tavrez commented 3 years ago

Weird, all the lines between init and GetForegroundWindow are just memory allocation and variable assignments. If you can, add a debug line between all the lines to be able to specify exactly which line is causing the crash.

On Sat, Jan 23, 2021 at 11:27 PM denniskniep notifications@github.com wrote:

Debug statements not showing up. I added further debug lines to make sure modifications are compiled

sk_enroll: START init_winhello: TEST v1 init_winhello: WARNING! This should not be like this! WinHello API Error: Is user available=0, User=0. client_converse: receive: unexpected internal error debug3: reap_helper: pid=13470 reap_helper: helper exited abnormally Key enrollment failed: unexpected internal error

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tavrez/openssh-sk-winhello/issues/8#issuecomment-766169916, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFM2DP76SOMK6AJ3FEOZXTS3MS4HANCNFSM4WPXHCMQ .

tavrez commented 3 years ago

Also please verify that your compiled file and openssh has same architecture(x86 or x64)

denniskniep commented 3 years ago

compiled file and openssh has same architecture (checked with file)

/usr/bin/ssh.exe: PE32 executable (console) Intel 80386, for MS Windows
winhello.dll: PE32 executable (DLL) (console) Intel 80386, for MS Windows

Added following debug statements:

        skdebug(__func__, "debug msg 4");
        if (init_winhello() != 0)
        {
                skdebug(__func__, "debug msg 4a");
                return SSH_SK_ERR_UNSUPPORTED;
        }

        skdebug(__func__, "debug msg 4b");

and

        skdebug(__func__, "TEST v1");
        if (isUserAvailable == 0 && user == 1)
                return 0;
        /* FIXME: As MS said, this should not happen, but it's happening! Contacted them but got no answer...
         * Related issue link: https://github.com/tavrez/openssh-sk-winhello/issues/1
         */
        skdebug(__func__, "WARNING! This should not be like this! WinHello API Error: Is user available=%d, User=%d.", isUserAvailable, user);
        return 0;

Result is:

debug1: sshsk_open: provider <path>/winhello.dll implements version 0x00070000
sk_enroll: START
sk_enroll: debug msg 1
sk_enroll: debug msg 2
sk_enroll: debug msg 3
sk_enroll: debug msg 4
init_winhello: TEST v1
init_winhello: WARNING! This should not be like this! WinHello API Error: Is user available=0, User=0.
client_converse: receive: unexpected internal error

....really weird

tavrez commented 3 years ago

I'll try to check it when I have more time. For now, you can try using the internal implementation of OpenSSH(be sure to run mobaXterm as administrator)

denniskniep commented 3 years ago

Thanks a lot for your help!

I have to make a correction: I used autoreconf --install during build and not autoconf --install as I mentioned above.

Sorry for that, but that should not make any trouble, right?

tavrez commented 3 years ago

autoreconf is correct, without it I think you can't compile at all.

denniskniep commented 3 years ago

Hi @tavrez, did you find some time to check whats the problem? Thanks

tavrez commented 3 years ago

Hey, I'm going to release a new version soon(small bug fixes). I'll check for this before releasing

tavrez commented 3 years ago

Could you tell me all the steps you made? (from installation of mobaXterm and required tools) I just compiled the code on cygwin without any problem :/

denniskniep commented 3 years ago

Could you tell me all the steps you made? (from installation of mobaXterm and required tools) sure, thanks for your support!

I tried the following setup on two different windows computers:

Install MobaXTerm

Download MobaXterm https://download.mobatek.net/2102021022292334/MobaXterm_Installer_v21.0.zip

Install MobaXterm

Start Session > Shell > Bash

uname –a CYGWIN_NT-10.0-WOW DESKTOP-ROV6A48 3.0.4(0.338/5/3) 2019-03-18 19:35 i686 GNU/Linux

ssh -V OpenSSH_7.5p1, OpenSSL 1.0.2o 27 Mar 2018

Install Dependencies to MobaXTerm Cygwin

Download Cygwin 32 bit Installer https://cygwin.com/setup-x86.exe

Execute Installer and choose: MobaXTerm cygwin root directory (C:\Users\<name\Documents\MobaXterm\slash) use mirror: https://linux.rz.ruhr-uni-bochum.de

Choose View: Full

Select Packages:

Name:openssh Version: 8.4p1-2
Name:make Version:4.3-1
Name:gcc-core Version:10.2.0-1
Name:libssl-devel Version: 1.1.1f-1
Name:libfido2 Version: 1.5.0-1
Name:libcbor Version:0.5.0-1

ssh -V OpenSSH_8.4p1, OpenSSL 1.1.1f 31 Mar 2020

Compile sk-winhello

Download https://github.com/tavrez/openssh-sk-winhello/releases/download/v2.0.0/winhello-2.0.0.tar.gz ./configure

make install (also copies winhello.dll to /usr/lib)

Execute keygen

Execute SSH_SK_PROVIDER=winhello.dll ssh-keygen -vvv -t ecdsa-sk -f ./fido

Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
debug3: start_helper: started pid=1404
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/sbin/ssh-sk-helper
debug1: sshsk_enroll: provider "winhello.dll", device "(null)", application "ssh:", userid "(null)", flags 0x01, challenge len 0
debug1: sshsk_enroll: using random challenge
debug1: sshsk_open: provider winhello.dll implements version 0x00070000
init_winhello: WARNING! This should not be like this! WinHello API Error: Is user available=0, User=0.
client_converse: receive: unexpected internal error
debug3: reap_helper: pid=1404
reap_helper: helper exited abnormally
Key enrollment failed: unexpected internal error

tavrez commented 3 years ago

Exception: STATUS_ACCESS_VIOLATION at eip=695C47F1
eax=00000000 ebx=695C1A0D ecx=80000008 edx=00000000 esi=FFFFFFF9 edi=00000000
ebp=0065CA7C esp=0065C9E4 program=C:\cygwin\usr\sbin\ssh-sk-helper.exe, pid 4199, thread main
cs=0023 ds=002B es=002B fs=0053 gs=002B ss=002B
Stack trace:
Frame     Function  Args
0065CA7C  695C47F1 (00000000, 00000000, 00000000, 00000000)
End of stack trace

alright, now I can reproduce. Trying to debug....

tavrez commented 3 years ago

Quick & dirty fix which worked for me: comment these lines: src/winhello.c#L87-L95

BOOL user = 0;
int isUserAvailable = webAuthNIsUserVerifyingPlatformAuthenticatorAvailable(&user);
if (isUserAvailable == 0 && user == 1)
    return 0;
skdebug(__func__, "WARNING! ...");

tavrez commented 3 years ago

the bug is a race condition inside 32 bit version of webAuthNIsUserVerifyingPlatformAuthenticatorAvailable, I need MS support to debug it, but I'm trying to go as much as I can

denniskniep commented 3 years ago

Thank you very much for your support and efforts!

tavrez commented 3 years ago

No prob, did that fix worked for you?

denniskniep commented 3 years ago

Now I am prompted for the FIDO Authenticator.

After touching the FIDO Authenticator there is a "unexpected internal error"

client_converse: receive: unexpected internal error
debug3: reap_helper: pid=9056
reap_helper: helper exited abnormally
Key enrollment failed: unexpected internal error

After drilling down with prints it seems that it is crashing now at this line: https://github.com/tavrez/openssh-sk-winhello/blob/4c316748a050cfaa9aff54505d5ece8d863920ec/src/winhello.c#L305

&pWebAuthNCredentialAttestation is printing with %p: 0x65ca18 pWebAuthNCredentialAttestation is printing with %p: 0xfffffff9

Same error on both of my workstations. Is it fully working for you and generating the private key?

Any ideas?

tavrez commented 3 years ago

May I know what kind of key are you using? This step is after key generation, means Winhello returned success without any issues, but I try to run multiple times to see if it's another race condition or not.

denniskniep commented 3 years ago

I use Yubico Security Key NFC https://www.yubico.com/de/product/security-key-nfc-by-yubico/

tavrez commented 3 years ago

Seems like the problem is related to fork in cygwin 32-bit apps.

Address space is a very limiting factor for Cygwin. These days, a full 32 bit Cygwin distro is not feasible anymore, and will in all likelihood fail in random places due to an issue with the fork(2) system call.

When I call Windows Hello in a simple app in 32-bit cygwin it works, but when I call it in OpenSSH(which use fork to call ssh-sk-helper and load my dll in that fork) it will fail, I can only suggest you to try to use 64-bit cygwin, and ask MobaXTerm developers about it.

tavrez / openssh-sk-winhello