Credential Manager doesn't work in Windows Containers with BusyBox MinGit

[dhirschfeld]

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

C:\> git --version --build-options
git version 2.16.1.windows.4
cpu: x86_64
built from commit: ef6d451bbfef86a529ebf12620289e0f15a93d8e

Running on a windowsservercore:1709 container:

C:\> (Get-CimInstance Win32_OperatingSystem).version
10.0.16299

C:\> cmd.exe /c ver

Microsoft Windows [Version 10.0.16299.192]

• What options did you set as part of the installation? Or did you choose the defaults?

I just unzipped the prebuilt release

• Any other interesting things about your environment that might be related to the issue you're seeing?

Running on a Windows Container

Details

• Running from powershell

  
  C:\> git clone https://github.stanwell.com/Trading/skeleton.git
  Cloning into 'skeleton'...
  bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor error: failed to execute prompt script (exit code 1) fatal: could not read Username for 'https://github.stanwell.com/Trading/skeleton.git': No error C:>

Note: The credential manager works correctly with the [full `.exe` installer](https://github.com/git-for-windows/git/releases/download/v2.16.1.windows.4/Git-2.16.1.4-64-bit.exe)

C:> git clone https://github.stanwell.com/Trading/skeleton.git Cloning into 'skeleton'... Logon failed, use ctrl+c to cancel basic credential prompt. Username for 'https://github.stanwell.com': dhirschfeld Password for 'https://dhirschfeld@github.stanwell.com': remote: Counting objects: 28, done. remote: Total 28 (delta 1), reused 1 (delta 1), pack-reused 26 Unpacking objects: 100% (28/28), done.

...*however* I'd like to user the BusyBox version as it fixes #1403/#1007 when running in a Windows Container.

What I'm hoping for is a git version which both allows cloning from a host-mapped volume *and* allows entering the username/password for the credential manager.

[dscho]

The full installer configures credential.helper = manager. What does this command tell you (with BusyBox-based MinGit)?

git config --show-origin credential.helper

I could imagine, though, that things would still fail with, say, multi-factor authorization, as I have no idea how the GitHub/Visual Studio Team Services/BitBucket/whatever helpers would behave if no graphical prompt was possible (IIRC most of them essentially show HTML pages): would they fall back to talking to the user via the console? Can they fall back that way?

The common strategy in Docker containers is to provide the credentials in different ways than via credential helpers: either use SSH keys, or configure the HTTPS remotes with credentials included. In your example, if the password was super-secret, then you would git clone https://dhirschfeld:super-secret@github.stanwell.com/Trading/skeleton.git.

(Of course, depending on your use case, you would want to ensure the password to be blotted out in whatever log your operations want to have.)

[dhirschfeld]

As shown in the working example at the bottom the credential manager just uses console input if you disallow the modal dialogue. This form of input is broken (bash tty error) though in the busybox version.

[dscho]

@dhirschfeld the Logon failed, use ctrl+c to cancel basic credential prompt message indeed comes from Git Credential Manager: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/blob/3a85f87a872891dedf90d5fee211d102a63eb8ac/Cli-CredentialHelper/Program.cs#L329

The message error: failed to execute prompt script (exit code 1), in contrast, comes directly from git.exe: https://github.com/git-for-windows/git/blob/ef6d451bbfef86a529ebf12620289e0f15a93d8e/compat/terminal.c#L141

This suggests, very strongly even, that your BusyBox example does not even try to call the Git Credential Manager.

I suspected this. That is the entire reason I asked you to provide the information what git config --show-origin credential.helper prints.

I still would highly encourage you to actually work with me to resolve this. So far, I feel that half of what I said was simply ignored, and I doubt that this will lead to resolving the issue.

If my suspicion is correct, and that git config call reports an empty value for the credential.helper setting, the issue could be resolved by calling git config --system credential.helper manager in the BusyBox-based Git. And in that case, I would also get something out of this ticket, as I would have to dig further into the scripts generating the BusyBox-based Git so that the Git Credential Manager can be configured by default.

[dhirschfeld]

Sorry @dscho - I sent my initial response from my phone on the way into work with the full intention of following up on your other points. Other priorities took precedence however and this fell off my radar. I am keen to get this resolved though so am looking into it now...

Also, thanks for taking the time to look into my problem - it is much appreciated! :)

[dhirschfeld]

It does indeed seem that the credential helper isn't set:

C:\> git config --show-origin credential.helper
C:\>

If I set the credential.helper though I still get the same result :(

C:\> git config --global credential.helper manager

C:\> git clone https://github.stanwell.com/Trading/skeleton.git
Cloning into 'skeleton'...
Logon failed, use ctrl+c to cancel basic credential prompt.
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error

C:\> git config --show-origin credential.helper
file:C:/Users/ContainerAdministrator/.gitconfig manager

[dhirschfeld]

Note: The comment below from here seems a bit ominous:

BusyBox-w32 has no idea (yet) about Git Bash's interactive terminal, and therefore does not support any interactive usage

Happy to test anything else out in my Container environment if you have any suggestions...

[dhirschfeld]

The common strategy in Docker containers is to provide the credentials in different ways than via credential helpers: either use SSH keys, or configure the HTTPS remotes with credentials included

My usecase is probably a bit different than the usual - I'm trying to configure a JupyterLab interactive analytics environment which is deployed on Windows Containers. So in this case I want my users to be able to provide their own credentials using the (basic) credential manger:

I am aware that you can embed your username and password in the remote url but I'm trying to avoid having to do that if at all possible

[dscho]

I sent my initial response from my phone on the way into work with the full intention of following up on your other points. Other priorities took precedence however and this fell off my radar. I am keen to get this resolved though so am looking into it now...

Thank you for apologizing. I know how this feels, some days (in fact, most of them!) feel like drowning in an ocean, just trying to survive.

If I set the credential.helper though I still get the same result :(

That is a real bummer. Could you set GIT_TRACE=1 before running the clone operation? For me, the output looks like this (and I get a nice GitHub login window):

D:\test-git\MinGit-2.16.1.3-BusyBox-64>set GIT_TRACE=1

D:\test-git\MinGit-2.16.1.3-BusyBox-64>cmd\git clone https://github.com/dscho/none
12:21:20.549203 git.c:344               trace: built-in: git 'clone' 'https://github.com/dscho/none'
Cloning into 'none'...
12:21:20.755657 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.com/dscho/none'
12:21:20.854183 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.com/dscho/none'
12:21:20.855184 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/dscho/none'
12:21:22.780909 run-command.c:627       trace: run_command: 'git credential-manager get'
12:21:23.023968 git.c:576               trace: exec: 'git-credential-manager' 'get'
12:21:23.024970 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'

Also, I wonder whether you see different behavior between running these commands in Jupyter as opposed to a CMD window?

[dhirschfeld]

The output with GIT_TRACE=1 is below:

C:\> $env:GIT_TRACE=1
C:\> git clone https://github.stanwell.com/Trading/skeleton.git
15:44:15.586700 git.c:344               trace: built-in: git 'clone' 'https://github.stanwell.com/Trading/skeleton.git'
Cloning into 'skeleton'...
15:44:15.619704 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
15:44:15.644699 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
15:44:15.645699 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
15:44:15.819711 run-command.c:627       trace: run_command: 'git credential-manager get'
15:44:16.093720 git.c:576               trace: exec: 'git-credential-manager' 'get'
15:44:16.094720 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
Logon failed, use ctrl+c to cancel basic credential prompt.
15:44:17.593774 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
15:44:17.596773 run-command.c:627       trace: run_command: 'C:\Miniconda3\Library\mingw64\bin\busybox.exe' '--help'
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error

The bash command is a pretty suspicious since bash (or WSL) isn't installed on the server.

[dhirschfeld]

I get the same results if I try directly from powershell in the container rather than through the JupyterLab/xterm.js terminal:

C:\> $env:USERNAME
ContainerAdministrator
C:\> git config --global credential.modalPrompt never
C:\> git config --global credential.helper manager
C:\> git config --show-origin credential.helper
file:C:/Users/ContainerAdministrator/.gitconfig manager
C:\> $env:GCM_INTERACTIVE='NEVER'
C:\> $env:GCM_PRESERVE_CREDS='TRUE'
C:\> $env:GIT_TRACE=1
C:\> $env:GCM_TRACE=1
C:\> git clone https://github.stanwell.com/Trading/skeleton.git
16:09:29.845751 git.c:344               trace: built-in: git 'clone' 'https://github.stanwell.com/Trading/skeleton.git'
Cloning into 'skeleton'...
16:09:29.892753 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:29.916748 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:29.917748 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:30.120757 run-command.c:627       trace: run_command: 'git credential-manager get'
16:09:30.238414 git.c:576               trace: exec: 'git-credential-manager' 'get'
16:09:30.239413 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
16:09:30.550416 ...\Common.cs:527       trace: [Main] git-credential-manager (v1.14.0) 'get'
16:09:30.761424 ...\Where.cs:240        trace: [FindGitInstallations] found 0 Git installation(s).
16:09:31.085437 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 4 entries.
16:09:31.094443 ...\Where.cs:240        trace: [FindGitInstallations] found 0 Git installation(s).
16:09:31.097441 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 4 entries.
16:09:31.122437 ...\Common.cs:395       trace: [LoadOperationArguments] GCM_INTERACTIVE = 'NEVER'.
16:09:31.123442 ...\Common.cs:437       trace: [LoadOperationArguments] preserve = 'True'.
16:09:31.678467 ...\Common.cs:77        trace: [CreateAuthentication] detecting authority type for 'https://github.stanwell.com/'.
16:09:31.696466 ...uthentication.cs:137 trace: [GetAuthentication] not github.com, authentication creation aborted.
16:09:31.698469 ...\Common.cs:172       trace: [CreateAuthentication] authority for 'https://github.stanwell.com/' is basic with NTLM=Auto.
16:09:31.721468 ...\Common.cs:565       trace: [QueryCredentials] credentials for 'https://github.stanwell.com/' not found.
16:09:31.722493 ...\Common.cs:495       trace: [LogEvent] Failed to retrieve credentials for 'https://github.stanwell.com/'.
16:09:31.767463 ...\Program.cs:265      trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
Logon failed, use ctrl+c to cancel basic credential prompt.
16:09:31.806468 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
16:09:31.810471 run-command.c:627       trace: run_command: 'C:\Miniconda3\Library\mingw64\bin\busybox.exe' '--help'
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error
C:\>

[dhirschfeld]

That is a real bummer.

Yeah, it would have been great if it was as simple as setting the credential manager! :smile:

I still hold out a glimmer of hope that something in the trace may jump out as an obvious/easy fix but it doesn't seem too likely :disappointed:

[dscho]

C:> $env:GCM_INTERACTIVE='NEVER'

Is this intentional? I would have expected this to tell the Git Credential Manager that it should not ask for the credentials...

[dscho]

The bash command is a pretty suspicious since bash (or WSL) isn't installed on the server.

It is included in MinGit (although not in BusyBox-based MinGit). And Git for Windows does not yet have code to special-case the BusyBox case and try a different tack on accessing the terminal in that case.

[dhirschfeld]

$env:GCM_INTERACTIVE='NEVER' appears to be necessary to force the git credential manager to use basic input i.e. to prompt for username and password at the console. Without that it just hangs, I believe because it tries to open a GUI dialog when there is no GUI on the containers.

Using the same configuration does work (prompts for username & password) with the full/normal git installer but then you run into #1403 so it's a case of pick your poison.

Git for Windows does not yet have code to special-case the BusyBox case and try a different tack on accessing the terminal in that case

I suspected that may be the case. I guess this issue can be left open as a reminder that it would be a useful feature to add to the BusyBox git.

[dscho]

$env:GCM_INTERACTIVE='NEVER' appears to be necessary to force the git credential manager to use basic input i.e. to prompt for username and password at the console. Without that it just hangs, I believe because it tries to open a GUI dialog when there is no GUI on the containers.

Using the same configuration does work (prompts for username & password) with the full/normal git installer but then you run into #1403 so it's a case of pick your poison.

Hmm, okay. My mistake, sorry for that.

Git for Windows does not yet have code to special-case the BusyBox case and try a different tack on accessing the terminal in that case

I suspected that may be the case. I guess this issue can be left open as a reminder that it would be a useful feature to add to the BusyBox git.

I'd rather fix the main problem first: ensure that the Win32 Console can be accessed by GCM even when it is called via BusyBox-based MinGit.

Let me try to set up something (will take a while, that's why I tried to avoid that so far, also because I am not altogether sure that I can reproduce the issue).

[dscho]

Let me try to set up something (will take a while, that's why I tried to avoid that so far, also because I am not altogether sure that I can reproduce the issue).

I can see the same issue you describe in a Powershell without Docker containers. Now, how do I debug this ;-)

[dscho]

Oy vey, I have been chasing the wrong goose all the time! This is what I get with non-BusyBox-backed MinGit:

[...]
15:28:28.539347 ...\Common.cs:142       trace: [CreateAuthentication] authority for 'https://github.com/' is GitHub.
15:28:28.566310 ...\Console.cs:47       trace: [CredentialPrompt] not a tty detected, abandoning prompt.
15:28:28.566817 ...uthentication.cs:224 trace: [InteractiveLogon] interactive logon for 'https://github.com/' failed.
15:28:28.567309 ...\Common.cs:666       trace: [QueryCredentials] credentials for 'https://github.com/' not found.
15:28:28.575862 ...\Common.cs:495       trace: [LogEvent] Failed to retrieve GitHub credentials for 'https://github.com/'.
15:28:28.590657 ...\Program.cs:265      trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
Logon failed, use ctrl+c to cancel basic credential prompt.
15:28:28.627658 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
Username for 'https://github.com':

Do you notice how it says [CredentialPrompt] not a tty detected, abandoning prompt and falls back to Git's own prompt, too? So the problem is not so much that Git Credential Manager fails to retrieve credentials from via terminal in BusyBox-based MinGit. It is more that it fails to retrieve credentials from terminal when Git was called from Powershell, whether BusyBox is involved or MinGit does not matter much. And the fall-back with Bash simply does not work with BusyBox, is all.

FWIW I found another environment variable you definitely want to set in your scenario:

$env:GCM_MODAL_PROMPT='No'

[dscho]

Good news! In a joint debugging session with @whoisj, we determined the root cause in Git Credential Manager. @dhirschfeld are you willing to test an experimental build (which I should have ready either later today, but definitely before noon tomorrow UTC+0100)?

Most likely I will also have a fix for the fallback code in Git at the same time.

[dscho]

credential-helper-w-tty-fix.zip

Okay, so this is the fixed Credential Manager. If you unpack this into your BusyBox-based MinGit's mingw64\libexec\git-core directory (overwriting existing files), it should start working.

[dhirschfeld]

Thanks for getting to the bottom of this @dscho! ...despite my efforts to confuse the issue with docker and busybox! :stuck_out_tongue_winking_eye:

I'll certainly test that out on my setup. May not get to it today but if not, will test it over the weekend...

[dhirschfeld]

FWIW I found another environment variable you definitely want to set in your scenario

I was instead using git config --global credential.modalPrompt never. That was the first thing I tried and was a little surprised that it wasn't sufficient to stop the dialog popping up. That's when I found out about the other environment variables which seemed to do the trick:

C:\> $env:GCM_INTERACTIVE='NEVER'
C:\> $env:GCM_PRESERVE_CREDS='TRUE'

If there's an equivalent git config for those I'd probably use that instead but I don't know that there is? I'll look into that, but it's not a problem if I do have to specify env vars to get the behaviour I'm after...

[dscho]

(This Git Credential Manager version was built from https://github.com/Microsoft/Git-Credential-Manager-for-Windows/pull/554.)

[dscho]

FWIW I found another environment variable you definitely want to set in your scenario

I was instead using git config --global credential.modalPrompt never. That was the first thing I tried and was a little surprised that it wasn't sufficient to stop the dialog popping up

I was surprised, too, that's why I studied the source code of Git Credential Manager and found GCM_MODAL_PROMPT.

[whoisj]

was instead using git config --global credential.modalPrompt never. That was the first thing I tried and was a little surprised that it wasn't sufficient to stop the dialog popping up.

That is likely because the only legal values are "true" or "false". 😏

Configuration Setting Names: ModalPrompt

[whoisj]

(This Git Credential Manager version was built from Microsoft/Git-Credential-Manager-for-Windows#554.)

Oh and for those wondering, this fix will be part of GCM v1.15.0 (releases sometime in March).

Thanks @dscho for the help, and thanks to @dhirschfeld for the inspiration to root cause and resolve the issue.

[dhirschfeld]

That is likely because the only legal values are true or false.

D'oh! Thanks for pointing that out @whoisj! I picked that up from an internet search. Probably should just have referred to the official documentation so thanks for the link! :)

[dhirschfeld]

@dscho - I did try patching git with the files in credential-helper-w-tty-fix.zip but it didn't seem sufficient. Maybe something else is required?

PS C:\> git clone https://github.stanwell.com/Trading/skeleton.git
Cloning into 'skeleton'...
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error

PS C:\> get-command git

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     git.exe                                            2.16.1.4   C:\temp\mingit\cmd\git.exe

PS C:\> ls C:\temp\mingit\mingw64\libexec\git-core\

    Directory: C:\temp\mingit\mingw64\libexec\git-core

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        2/15/2018  10:40 AM                mergetools
-a----        2/23/2018   8:49 AM         325120 Bitbucket.Authentication.dll
-a----        1/12/2018   7:06 AM         219168 git-askpass.exe
-a----         2/8/2018   3:59 AM          14444 git-bisect
-a----        2/23/2018   8:49 AM         158720 git-credential-manager.exe
-a----         2/8/2018   3:59 AM          15543 git-filter-branch
-a----         2/8/2018   3:59 AM           2477 git-merge-octopus
-a----         2/8/2018   3:59 AM           3707 git-merge-one-file
-a----         2/8/2018   3:59 AM            944 git-merge-resolve
-a----         2/8/2018   3:59 AM          10303 git-mergetool
-a----         2/8/2018   3:59 AM           8869 git-mergetool--lib
-a----         2/8/2018   3:59 AM           2650 git-parse-remote
-a----         2/8/2018   3:59 AM           3596 git-quiltimport
-a----         2/8/2018   3:59 AM          16892 git-rebase
-a----         2/8/2018   3:59 AM           2597 git-rebase--am
-a----         2/8/2018   3:59 AM          28506 git-rebase--interactive
-a----         2/8/2018   3:59 AM           3933 git-rebase--merge
-a----         2/8/2018   3:59 AM           3698 git-request-pull
-a----         2/8/2018   3:59 AM           2381 git-sh-i18n
-a----         2/8/2018   3:59 AM           9676 git-sh-setup
-a----         2/8/2018   3:59 AM          15486 git-stash
-a----         2/8/2018   3:59 AM          26818 git-submodule
-a----         2/8/2018   3:59 AM          16400 git-subtree
-a----         2/6/2018  10:30 PM            217 git-update
-a----         2/8/2018   3:59 AM           4401 git-web--browse
-a----        2/23/2018   8:49 AM         299008 GitHub.Authentication.exe
-a----        2/23/2018   8:48 AM          47616 Microsoft.Alm.Authentication.dll
-a----        2/23/2018   8:48 AM          29696 Microsoft.Alm.Git.dll
-a----        3/22/2017   7:08 AM         204576 Microsoft.IdentityModel.Clients.ActiveDirectory.dll
-a----        3/22/2017   7:08 AM          89912 Microsoft.IdentityModel.Clients.ActiveDirectory.Platform.dll
-a----        2/23/2018   8:48 AM          76288 Microsoft.Vsts.Authentication.dll

[dhirschfeld]

With $env:GIT_TRACE=1:

PS C:\> git clone https://github.stanwell.com/Trading/skeleton.git
17:33:41.222368 git.c:344               trace: built-in: git 'clone' 'https://github.stanwell.com/Trading/skeleton.git'
Cloning into 'skeleton'...
17:33:41.262369 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
17:33:41.288367 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
17:33:41.289367 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
17:33:41.493378 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
17:33:41.496377 run-command.c:627       trace: run_command: 'C:\temp\mingit\mingw64\bin\busybox.exe' '--help'
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error

Note: $env:GCM_TRACE=1 didn't seem to have an effect.

[dhirschfeld]

Note: This was using a mingit from a host-mapped volume (C:/temp). I don't think that should affect anything though?

[dscho]

@dhirschfeld earlier, you got this:

[...]
16:09:29.917748 run-command.c:627 trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:30.120757 run-command.c:627 trace: run_command: 'git credential-manager get'
[...]

But this time, you got

[...]
17:33:41.289367 run-command.c:627 trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
17:33:41.493378 run-command.c:627 trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
[...]

This makes me believe that this (host-mapped) MinGit does not have credential.helper configured... Is my hunch correct?

[dhirschfeld]

Sorry @dscho - dumb mistake.

After setting git config --global credential.helper manager though it still doesn't seem to work:

PS C:\> git clone https://github.stanwell.com/Trading/skeleton.git
09:56:06.979782 git.c:344               trace: built-in: git 'clone' 'https://github.stanwell.com/Trading/skeleton.git'
Cloning into 'skeleton'...
09:56:07.013783 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
09:56:07.040784 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
09:56:07.041785 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
09:56:07.215792 run-command.c:627       trace: run_command: 'git credential-manager get'
09:56:07.270792 git.c:576               trace: exec: 'git-credential-manager' 'get'
09:56:07.270792 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
09:56:08.172827 ...\Common.cs:636       trace: [Main] git-credential-manager (v1.15.0) 'get'
09:56:08.314831 ...\Where.cs:236        trace: [FindGitInstallations] found 0 Git installation(s).
09:56:08.498836 ...Configuration.cs:218 trace: [LoadGitConfiguration] git All config read, 3 entries.
09:56:08.514837 ...\Where.cs:236        trace: [FindGitInstallations] found 0 Git installation(s).
09:56:08.516836 ...Configuration.cs:218 trace: [LoadGitConfiguration] git All config read, 3 entries.
09:56:08.545839 ...\Common.cs:431       trace: [LoadOperationArguments] interactive = 'NEVER'.
09:56:08.545839 ...\Common.cs:465       trace: [LoadOperationArguments] modalPrompt = 'False'.
09:56:08.545839 ...\Common.cs:484       trace: [LoadOperationArguments] GCM_PRESERVE_CREDS = ''.
09:56:08.546839 ...\Common.cs:488       trace: [LoadOperationArguments] WARNING: the 'GCM_PRESERVE_CREDS' variable has been deprecated, use 'preserve' instead.
09:56:08.661849 ...\Common.cs:80        trace: [CreateAuthentication] detecting authority type for 'https://github.stanwell.com/'.
09:56:08.681852 ...uthentication.cs:135 trace: [GetAuthentication] not github.com, authentication creation aborted.
09:56:08.682853 ...\Common.cs:178       trace: [CreateAuthentication] authority for 'https://github.stanwell.com/' is basic with NTLM=Auto.
09:56:08.706851 ...\Common.cs:679       trace: [QueryCredentials] credentials for 'https://github.stanwell.com/' not found.
09:56:08.706851 ...\Common.cs:601       trace: [LogEvent] Failed to retrieve credentials for 'https://github.stanwell.com/'.
09:56:08.738850 ...\Program.cs:330      trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
Logon failed, use ctrl+c to cancel basic credential prompt.
09:56:08.759850 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
09:56:08.763849 run-command.c:627       trace: run_command: 'C:\temp\mingit\mingw64\bin\busybox.exe' '--help'
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error
PS C:\>

Listing the git-core folder shows that the binaries patched on 23rd Feb are there, as shown previously.

[dhirschfeld]

Note: My testing is from powershell on a windowsservercore 1709 container in case that makes a difference.

[dscho]

Hmm. That's really strange. And dropping that GCM build into a portable Git still works in the same Powershell?

[dhirschfeld]

The files in C:\temp\mingit\mingw64\libexec\git-core were overwritten with those in credential-helper-w-tty-fix.zip but I still got the bash: Unhandled device /dev/tty error

Maybe it's something to do with the manual patching? I can test the next release on my setup to either confirm or refute that.

[dscho]

Hmmm. I just unpacked https://github.com/git-for-windows/git/releases/download/v2.16.2.windows.1/MinGit-2.16.2-busybox-64-bit.zip, overwrote all 8 Git Credential Manager files in mingw64\libexec\git-core with the ones from credential-helper-w-tty-fix.zip and it worked:

PS C:\Users\me> $env:GCM_MODAL_PROMPT='No'
PS C:\Users\me> $env:GCM_INTERACTIVE='Always'
PS C:\Users\me> $env:GIT_TRACE=1
PS C:\Users\me> D:\test-git\MinGit-2.16.2-BusyBox-64\cmd\git ls-remote https://github.com/dscho/non-existent
11:22:28.834736 git.c:344               trace: built-in: git 'ls-remote' 'https://github.com/dscho/non-existent'
11:22:28.836741 run-command.c:627       trace: run_command: 'remote-https' 'https://github.com/dscho/non-existent' 'https://github.com/dscho/non-existent'
11:22:28.903769 git.c:576               trace: exec: 'git-remote-https' 'https://github.com/dscho/non-existent' 'https://github.com/dscho/non-existent'
11:22:28.904769 run-command.c:627       trace: run_command: 'git-remote-https' 'https://github.com/dscho/non-existent' 'https://github.com/dscho/non-existent'
11:22:30.156928 run-command.c:627       trace: run_command: 'git credential-manager get'
11:22:30.325428 git.c:576               trace: exec: 'git-credential-manager' 'get'
11:22:30.326427 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
Please enter your GitHub credentials for https://github.com/
username:

Then I unpacked MinGit again, overwriting everything in D:\test-git\MinGit-2.16.2-BusyBox-64 and ran it again:

PS C:\Users\me> D:\test-git\MinGit-2.16.2-BusyBox-64\cmd\git ls-remote https://github.com/dscho/non-existent
11:25:35.676070 git.c:344               trace: built-in: git 'ls-remote' 'https://github.com/dscho/non-existent'
11:25:35.678056 run-command.c:627       trace: run_command: 'remote-https' 'https://github.com/dscho/non-existent' 'https://github.com/dscho/non-existent'
11:25:35.740071 git.c:576               trace: exec: 'git-remote-https' 'https://github.com/dscho/non-existent' 'https://github.com/dscho/non-existent'
11:25:35.741071 run-command.c:627       trace: run_command: 'git-remote-https' 'https://github.com/dscho/non-existent' 'https://github.com/dscho/non-existent'
11:25:36.966998 run-command.c:627       trace: run_command: 'git credential-manager get'
11:25:37.149998 git.c:576               trace: exec: 'git-credential-manager' 'get'
11:25:37.151000 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
Logon failed, use ctrl+c to cancel basic credential prompt.
11:25:37.519093 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
Username for 'https://github.com':

There is a peculiar thing going on here that I probably should explain: I do have an early WSL version installed and therefore bash will execute C:\Windows\system32\bash.exe, which in my setup was originally unable to access /dev/tty, but once I tested with a portable Git (and consequently, with the MSYS2 bash.exe included in the portable Git, which could access /dev/tty) all of a sudden the WSL bash.exe started to also be able to access /dev/tty.

So the short version is: in my hands, the w-tty-fix work as expected here, while vanilla MinGit-2.16.2-busybox' GCM binaries don't.

So I went back to study the log you provided and I saw this:

[...]
09:56:08.661849 ...\Common.cs:80 trace: [CreateAuthentication] detecting authority type for 'https://github.stanwell.com/'.
09:56:08.681852 ...uthentication.cs:135 trace: [GetAuthentication] not github.com, authentication creation aborted.
09:56:08.682853 ...\Common.cs:178 trace: [CreateAuthentication] authority for 'https://github.stanwell.com/' is basic with NTLM=Auto.
09:56:08.706851 ...\Common.cs:679 trace: [QueryCredentials] credentials for 'https://github.stanwell.com/' not found.
09:56:08.706851 ...\Common.cs:601 trace: [LogEvent] Failed to retrieve credentials for 'https://github.stanwell.com/'.
09:56:08.738850 ...\Program.cs:330 trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
[...]

and this reminded me that I had forgotten to set $env:GCM_TRACE = 1. The relevant logs from re-testing are:

[... GCM 1.14.0 ...]
11:31:05.493365 ...\Common.cs:77        trace: [CreateAuthentication] detecting authority type for 'https://github.com/'.
11:31:05.509368 ...uthentication.cs:132 trace: [GetAuthentication] created GitHub authentication for 'https://github.com/'.
11:31:05.514376 ...\Common.cs:142       trace: [CreateAuthentication] authority for 'https://github.com/' is GitHub.
11:31:05.530387 ...\Console.cs:47       trace: [CredentialPrompt] not a tty detected, abandoning prompt.
11:31:05.531389 ...uthentication.cs:224 trace: [InteractiveLogon] interactive logon for 'https://github.com/' failed.
11:31:05.536384 ...\Common.cs:666       trace: [QueryCredentials] credentials for 'https://github.com/' not found.
11:31:05.537385 ...\Common.cs:495       trace: [LogEvent] Failed to retrieve GitHub credentials for 'https://github.com/'.
11:31:05.542387 ...\Program.cs:265      trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
[...]

and

[... patched GCM ...]
11:40:29.427590 ...\Common.cs:80        trace: [CreateAuthentication] detecting authority type for 'https://github.com/'.
11:40:29.444590 ...uthentication.cs:130 trace: [GetAuthentication] created GitHub authentication for 'https://github.com/'.
11:40:29.444590 ...\Common.cs:147       trace: [CreateAuthentication] authority for 'https://github.com/' is GitHub.
11:40:29.465591 ...\Console.cs:74       trace: [CredentialPrompt] console mode = '499'.
Please enter your GitHub credentials for https://github.com/
username:

And this, in turn, makes me appreciate a crucial difference in our testing: I tested against a github.com URL with a non-existent (and therefore possibly private) repository, while you tested against github.stanwell.com (which my machine cannot resolve to an IP).

The error message "not github.com" in particular comes from here: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/blob/71862fb1e96acb70053a99acd683e5a3279dc5fe/GitHub.Authentication/Authentication.cs#L126-L136, and that code points to the obvious suffix test for "github.com" in the host part of the URL: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/blob/master/GitHub.Authentication/Authentication.cs#L38

Which means that this probably never worked in GCM for you, but that the prompt you saw with portable Git was actually from the fall-back (which I fixed in https://github.com/git-for-windows/git/pull/1514).

@dhirschfeld Can you confirm that the "not github.com" error message shows up in your testing when you run it with a portable Git, and that it does fall back to running that Bash snippet that communicates via /dev/tty?

Also, for my curiosity: could you clarify what github.stanwell.com refers to? Is this a private GitHub instance? @whoisj is this supposed to work with GCM?

[dscho]

Okay, so I tried with another tack to reproduce your scenario: I set up an Apache instance with a self-signed certificate and requiring basic authentication. But the relevant part of my trace looks like this:

[...]
12:34:18.511435 ...\Common.cs:80        trace: [CreateAuthentication] detecting authority type for 'https://127.0.0.1/'.
12:34:18.530436 ...uthentication.cs:135 trace: [GetAuthentication] not github.com, authentication creation aborted.
12:34:18.531435 ...\Common.cs:178       trace: [CreateAuthentication] authority for 'https://127.0.0.1/' is basic with NTLM=Auto.
12:34:18.675936 ...enticateHelper.cs:71 trace: [GetHeaderValues] error testing targetUri for NTML: An error occurred while sending the request.
12:34:18.678438 ...uthentication.cs:125 trace: [AcquireCredentials] prompting user for credentials for 'https://127.0.0.1/'.
12:34:18.685936 ...\Console.cs:74       trace: [CredentialPrompt] console mode = '503'.
Please enter your credentials for https://127.0.0.1/
username:

So: again, it does what I wanted it to do here...

But then, my system is not set up for NTLM, and your github.stanwell.com probably is?

[dhirschfeld]

Our GitHub Enterprise uses LDAP authentication so I think that's probably NTLM?

I'll try with github.com tomorrow and post the results...

[whoisj]

LDAP authentication so I think that's probably NTLM?

Probably, doesn't have to be. Auth is complicated. 🤕

[dscho]

09:56:08.545839 ...\Common.cs:431 trace: [LoadOperationArguments] interactive = 'NEVER'.

Oh, right. I chatted to @whoisj and this means that GCM must not ask you anything.

@dhirschfeld Can you try again with $env:GCM_INTERACTIVE = 'auto'?

[dhirschfeld]

@dscho - that was the trick! Setting $env:GCM_INTERACTIVE='auto' worked with the patched files. Thanks for your perseverance in getting to the bottom of this!

[dhirschfeld]

In case it's of use this is the dockerfile I used to create a test container:

#  docker build --no-cache --isolation hyperv -t test-gcm

FROM microsoft/windowsservercore

SHELL ["powershell", "-Command", "$ErrorActionPreference='Stop'; $ProgressPreference='SilentlyContinue';"]

ADD https://github.com/git-for-windows/git/releases/download/v2.16.2.windows.1/MinGit-2.16.2-busybox-64-bit.zip C:/
ADD https://github.com/git-for-windows/git/files/1749900/credential-helper-w-tty-fix.zip C:/

WORKDIR C:/

RUN Expand-Archive -Path 'C:/MinGit-2.16.2-busybox-64-bit.zip' -DestinationPath 'C:/git' -Force; \
    Expand-Archive -Path 'C:/credential-helper-w-tty-fix.zip' -DestinationPath 'C:/git/mingw64/libexec/git-core' -Force; \
    setx /M PATH $($env:PATH + ';' + 'C:\git\cmd;C:\git\mingw64\bin');
RUN Start-Process git -ArgumentList 'config','--global','http.sslVerify','false' -NoNewWindow -Wait; \
    Start-Process git -ArgumentList 'config','--global','credential.modalPrompt','false' -NoNewWindow -Wait; \
    Start-Process git -ArgumentList 'config','--global','credential.helper','manager' -NoNewWindow -Wait;

ENV GCM_INTERACTIVE='NEVER'
ENV GCM_PRESERVE_CREDS='TRUE'

SHELL ["cmd", "/S", "/C"]

docker run --isolation=hyperv -it test-gcm powershell

[dhirschfeld]

I've also pushed my test-gcm image to DockerHub - https://hub.docker.com/r/dhirschfeld/test-gcm/

[whoisj]

@dhirschfeld just for the record this is potentially unsafe:

Start-Process git -ArgumentList 'config','--global','http.sslVerify','false' -NoNewWindow -Wait;

and thanks for this:

I've also pushed my test-gcm image to DockerHub - https://hub.docker.com/r/dhirschfeld/test-gcm/

[dhirschfeld]

http.sslVerify=false was just for testing for now.

I do have to check though if the busybox git by default uses the Windows Certificate Store - i.e. when using the gui installer I select the native Secure Channel Library:

Maybe @dscho knows the answer to that? Or if there's a way to check/configure post-install? (which for busybox git is just unzip archive)

[whoisj]

@dhirschfeld look for http.sslbackend=schannel from git-config.

[dscho]

@dhirschfeld thanks for posting the Dockerfile, and for pushing the image!

look for http.sslbackend=schannel from git-config.

Indeed, that is all the installer does: set that config setting in the "system-wide" (i.e. Git for Windows-wide) config.