git push hangs when GIT_SSH is OpenSSH

AnthonyMastrean commented 5 years ago

"OpenSSH for Windows" version

PS> ((Get-Item (Get-Command sshd).Source).VersionInfo.FileVersion)
7.7.2.2

Client OperatingSystem

PS> ((Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows nt\CurrentVersion\" -Name ProductName).ProductName)
Windows 10 Pro

What is failing

When GIT_SSH is set to OpenSSH, per the advice here https://github.com/dahlbyk/posh-git/issues/575#issuecomment-391386119, seemingly any git push hangs indefinitely? I had to send an interrupt (CTRL+C) to get the console back.

GIT_SSH=C:\WINDOWS\System32\OpenSSH\ssh.exe

Expected output

A successful push 🤗

Actual output

$ GIT_TRACE=1 git push --force-with-lease
16:04:28.047029 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
16:04:28.050024 git.c:419               trace: built-in: git push --force-with-lease
16:04:28.068027 run-command.c:643       trace: run_command: unset GIT_PREFIX; 'C:\WINDOWS\System32\OpenSSH\ssh.exe' git@gitlab.com 'git-receive-pack '\''vaeit/nextgen/packaging.git'\'''
16:04:29.069626 run-command.c:643       trace: run_command: .git/hooks/pre-push origin git@gitlab.com:vaeit/nextgen/packaging.git
16:04:29.217805 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
16:04:29.219805 git.c:676               trace: exec: git-lfs pre-push origin git@gitlab.com:vaeit/nextgen/packaging.git
16:04:29.220805 run-command.c:643       trace: run_command: git-lfs pre-push origin git@gitlab.com:vaeit/nextgen/packaging.git
16:04:29.249803 trace git-lfs: exec: git 'version'
16:04:29.294969 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
16:04:29.380835 trace git-lfs: exec: git 'config' '-l'
16:04:29.403839 trace git-lfs: pre-push: refs/heads/INFRA-489 47d6d3948f7a15c31e9c019256d2735e452195f6 refs/heads/INFRA-489 963168ae70b8ac4fc406fc1440b2bb9f12a77630
16:04:30.887486 trace git-lfs: run_command: C:\WINDOWS\System32\OpenSSH\ssh.exe -- git@gitlab.com git-lfs-authenticate vaeit/nextgen/packaging.git upload
Connection reset by 35.231.145.151 port 22
16:07:26.575373 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
16:07:26.575373 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
16:07:26.575373 trace git-lfs: filepathfilter: accepting "tmp"

Exiting because of "interrupt" signal.

rkeithhill commented 5 years ago

One possibility is that GitLab is using a cipher suite that is not configured by default with OpenSSH. I ran into this issue with TFS Git on our TFS 2015 server. Try this to enable verbose logging:

$env:GIT_SSH_COMMAND = '"C:\Windows\System32\OpenSSH\ssh.exe" -vvv'

Try the push and see what you get in the output.

seed-of-apricot commented 5 years ago

I ran into the same issue, with multiple SSH keys managed. The problem is, I believe, in git lfs because when I don't install lfs it works perfectly.

Related issue

obrienmd commented 5 years ago

I see this with repos with git-lfs as well, but no others. Turning on ssh -vvv and git trace provides additional clues. The initial ssh connection works and connects as expected, but the process gets stuck here with no additional debug output:

debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
14:16:36.985296 run-command.c:663       trace: run_command: .git/hooks/pre-push origin git@a.gitserver.com:mygroup/my-repo.git
14:16:37.171576 exec-cmd.c:238          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:16:37.196030 git.c:704               trace: exec: git-lfs pre-push origin git@a.gitserver.com:mygroup/my-repo.git
14:16:37.197028 run-command.c:663       trace: run_command: git-lfs pre-push origin git@a.gitserver.com:mygroup/my-repo.git
14:16:38.022938 trace git-lfs: exec: git 'version'
14:16:38.119177 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
14:16:38.474769 trace git-lfs: exec: git 'config' '-l'
14:16:39.907192 trace git-lfs: run_command: sh -c "C:\Windows\System32\OpenSSH\ssh.exe" -vvv -- git@a.gitserver.com 'git-lfs-authenticate mygroup/my-repo.git upload'

I think it's able to connect just fine, but when the run_command is dispatched it hangs.

Same openssh and Win10 versions as original poster. Other versions:

git-lfs/2.7.2 (GitHub; windows amd64; go 1.12.2; git 08a08ae0)
git version 2.22.0.windows.1
GIT_EXEC_PATH=C:/Program Files/Git/mingw64/libexec/git-core
GIT_LFS_PATH=C:\Program Files\Git LFS
GIT_SSH_COMMAND="C:\Windows\System32\OpenSSH\ssh.exe" -vvv
GIT_TRACE=1

lephyrus commented 5 years ago

I'm having the exact same issue. The trace looks the same.

==== snip ====
11:27:30.147136 trace git-lfs: pre-push: refs/heads/nvm-test e6df7fd73686a4a16617f5b4ddeca75c37abf625 refs/heads/nvm-test 0000000000000000000000000000000000000000
11:27:31.818155 trace git-lfs: run_command: C:\Users\REDACTED\scoop\apps\win32-openssh\current\ssh.exe -- git@github.com git-lfs-authenticate myorg/myrepo.git upload
=== hangs here, output below after CTRL-C ===
11:30:27.533441 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"
11:30:27.533441 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
11:30:27.539438 trace git-lfs: filepathfilter: accepting "tmp"
11:30:27.541438 trace git-lfs: filepathfilter: accepting "objects"

I'd like to add that running the command which appears to hang manually is successful:

> C:\Users\REDACTED\scoop\apps\win32-openssh\current\ssh.exe -- git@github.com git-lfs-authenticate myorg/myrepo.git upload
{
  "href": "https://lfs.github.com/myorg/myrepo.git",
  "header": {
    "Authorization": "RemoteAuth REDACTED"
  },
  "expires_at": "2019-06-20T13:56:42Z",
  "expires_in": 21599
}

Versions:

OpenSSH_for_Windows_7.9p1, LibreSSL 2.6.5
git-lfs/2.7.2 (GitHub; windows amd64; go 1.12.2; git 08a08ae0)
git version 2.22.0.windows.1

davclark commented 5 years ago

I have the same issue with the same configuration. I tried with git installed via scoop and also via chocolatey. I'm trying now using the openssh installed from scoop. That works fine... it's a minor annoyance for someone who wants a clean install, since you've got two versions of OpenSSH installed, and the scoop version installs the whole msys2 stack. But having a working git is more important! Looking forward to a cleaner fix.

I think this establishes that it's an issue with the microsoft-packaged OpenSSH?

yogasantosa commented 5 years ago

Latest release (v8.0.0.0p1-Beta) from https://github.com/PowerShell/Win32-OpenSSH/releases works fine.

The one activated through Windows 10 optional feature is as far as i know - is 7.9 and having this hanging issue.

AnthonyMastrean commented 5 years ago

@yogasantosa do you see anything specific that fixes this issue (in the linked milestone, maybe)? Or does this one just happen to work in your testing?

yogasantosa commented 5 years ago

@AnthonyMastrean - found this by accident:

got 2 machines: 1 running Windows 10, the other windows 7
in windows 10, i used optional feature to install (v 7.9)
in windows 7, i installed from the link I shared (v 8.0)
and i noticed in windows 10 machine it was hanging, but not on windows 7
then i checked the version, uninstalled the one on windows 10 and installed v8.0 and it works

Looking at the milestone there seemed to be two possible issues related to this one

maertendMSFT commented 4 years ago

This has been reported to be working, please reopen if this repros on the latest release.

PowerShell / Win32-OpenSSH

git push hangs when GIT_SSH is OpenSSH #1377