SMillerDev
commented
1 year ago Seems to be similar to https://github.com/openssl/openssl/issues/18420
Closed ffloimair closed 1 year ago
SMillerDev
commented
1 year ago Seems to be similar to https://github.com/openssl/openssl/issues/18420
ffloimair
commented
1 year ago Not necessarily. While it is related as Azure DevOps uses an old Host-Key Algorithm, it doesn't explain the segfault. The config override was already necessary in earlier versions of openssh. This seems to be related to the use of openssl 3.0 as default now. Possibly this openSSL version deprecated this behaviour altogether, but still this should not segfault.
What is even more dubious is the fact that this happens in combination with git, without git giving you even a hint what the problem is. It took me several hours to narrow down that the problem is in fact within openssh.
Whatever this leads to it still should not segfault but instead show an error that is descriptive of the situation. If the solution to get it working is pinning openssh for those who need it, I am okay with that, but still a segfault is a no-go.
SMillerDev
commented
1 year ago Can you run this command with a debugger to see what is happening with the segfault?
ffloimair
commented
1 year ago I'll see if I can find time to do so either tonight or tomorrow. Please give me some time.
Bo98
commented
1 year ago Thanks for the detailed steps for reproduction. I've been unable to reproduce the issue so far:
$ ssh -Tv git@ssh.dev.azure.com
OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023
debug1: Reading configuration data /Users/bo/.ssh/config
debug1: /Users/bo/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to ssh.dev.azure.com [51.104.26.0] port 22.
debug1: Connection established.
debug1: identity file /Users/bo/.ssh/id_rsa type -1
debug1: identity file /Users/bo/.ssh/id_rsa-cert type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/bo/.ssh/id_ed25519 type -1
debug1: identity file /Users/bo/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/bo/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/bo/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/bo/.ssh/id_xmss type -1
debug1: identity file /Users/bo/.ssh/id_xmss-cert type -1
debug1: identity file /Users/bo/.ssh/id_dsa type -1
debug1: identity file /Users/bo/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3
debug1: Remote protocol version 2.0, remote software version SSHBlackbox.10
debug1: compat_banner: no match: SSHBlackbox.10
debug1: Authenticating to ssh.dev.azure.com:22 as 'git'
debug1: load_hostkeys: fopen /Users/bo/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_GROUP received
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: SSH2_MSG_KEX_DH_GEX_REPLY received
debug1: Server host key: ssh-rsa SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og
debug1: load_hostkeys: fopen /Users/bo/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.dev.azure.com' is known and matches the RSA host key.
debug1: Found key in /Users/bo/.ssh/known_hosts:102
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /Users/bo/.ssh/id_rsa
debug1: Will attempt key: /Users/bo/.ssh/id_ecdsa
debug1: Will attempt key: /Users/bo/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/bo/.ssh/id_ed25519
debug1: Will attempt key: /Users/bo/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/bo/.ssh/id_xmss
debug1: Will attempt key: /Users/bo/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: password,publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/bo/.ssh/id_rsa
debug1: Trying private key: /Users/bo/.ssh/id_ecdsa
debug1: Trying private key: /Users/bo/.ssh/id_ecdsa_sk
debug1: Trying private key: /Users/bo/.ssh/id_ed25519
debug1: Trying private key: /Users/bo/.ssh/id_ed25519_sk
debug1: Trying private key: /Users/bo/.ssh/id_xmss
debug1: Trying private key: /Users/bo/.ssh/id_dsa
debug1: Next authentication method: password
git@ssh.dev.azure.com's password:
interesting
cho-m
commented
1 year ago I can reproduce a segfault using x86_64 bottle on Rosetta installation (though I didn't set up RSA key for Azure or SSH configs so can't guarantee exact same situation). I will note that I can't reproduce with ARM bottle installation though with same setup/configs.
Process 11772 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
frame #0: 0x000000010910019a libcrypto.3.dylib`BN_num_bits + 17
libcrypto.3.dylib`BN_num_bits:
-> 0x10910019a <+17>: movslq 0x8(%rdi), %rcx
0x10910019e <+21>: leal -0x1(%rcx), %r13d
0x1091001a2 <+25>: testb $0x4, 0x14(%rdi)
0x1091001a6 <+29>: jne 0x1091001c6 ; <+61>
Target 0: (ssh) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
* frame #0: 0x000000010910019a libcrypto.3.dylib`BN_num_bits + 17
frame #1: 0x000000010005bbbe ssh`___lldb_unnamed_symbol871 + 78
frame #2: 0x00000001000e3149 ssh`___lldb_unnamed_symbol1817 + 537
frame #3: 0x00000001000e4d99 ssh`___lldb_unnamed_symbol1830 + 1209
frame #4: 0x00000001000a5506 ssh`___lldb_unnamed_symbol1372 + 390
frame #5: 0x00000001000a566a ssh`___lldb_unnamed_symbol1373 + 42
frame #6: 0x000000010002fba4 ssh`___lldb_unnamed_symbol588 + 1220
frame #7: 0x000000010002cf1f ssh`___lldb_unnamed_symbol572 + 335
frame #8: 0x0000000100009856 ssh`___lldb_unnamed_symbol437 + 21478
frame #9: 0x000000020014541f dyld`start + 1903Looking at recompiled symbols, in my case, kexgex_hash is called with NULL values (e.g. prime=0x0000000000000000) at https://github.com/openssh/openssh-portable/blob/master/kexgexc.c#L187-L198
carlocab
commented
1 year ago I can reproduce this with the Ventura bottle but not the Monterey one. Using the Monterey bottle for now might be enough of a workaround:
brew fetch --bottle-tag=monterey openssh
brew uninstall openssh
brew install "$(brew --cache --bottle-tag=monterey openssh)"We could try building with llvm instead.
I think part of issue is OpenSSH's configure.ac Clang version-specific logic, e.g.:
# clang 15 seems to have a bug in -fzero-call-used-regs=all. See
# https://bugzilla.mindrot.org/show_bug.cgi?id=3475 and
# https://github.com/llvm/llvm-project/issues/59242
case "$CLANG_VER" in
15.*) OSSH_CHECK_CFLAG_COMPILE([-fzero-call-used-regs=used]) ;;
*) OSSH_CHECK_CFLAG_COMPILE([-fzero-call-used-regs=all]) ;;
esacXcode 14.3.1 (Apple clang 14.0.3) is LLVM 15. Also, the version detection doesn't work with Apple Clang (nor with Homebrew LLVM's Clang) and returns:
checking clang version... versionThe "Apple"/"Homebrew" in clang -v causes issues (and wouldn't help detect upstream version for Apple).
cho-m
commented
1 year ago Trying out an inreplace hack in #135373. Didn't see any issue when attempted locally but haven't checked if this is the root cause of issue.
ffloimair
commented
1 year ago openssh_9.3p1_2 bottle fixes the issue on my machine. Thanks @cho-m !!!
j0k3r
commented
1 year ago Problem fixed for me as well, no more segfault. 👏🏼
brew gist-logs <formula>link ORbrew configANDbrew doctoroutputbrew gist-logs openssh
Error: No logs.brew config
brew doctor
Verification
brew doctoroutput" saysYour system is ready to brew.and am still able to reproduce my issue.brew updateand am still able to reproduce my issue.brew doctorand that did not fix my problem.What were you trying to do (and why)?
git fetch from an Azure DevOps Repository
What happened (include all command output)?
What did you expect to happen?
data can be fetched from remote repository
However this fails
Doing a cross-test with ssh directly i was able to reproduce what is the problem: openssh segfaults
If i revert openssh to commit 2848b48f4b26e8d568c6514aff9cdfb4c6849ef0 (openssh 9.3p1), install the formula and pin it. Everything works fine (both ssh test and git).
Step-by-step reproduction instructions (by running
brewcommands)git clone
or
git fetch (if the repo is already cloned locally)
will fail
If you do the crosstest with ssh (no Azure DevOps account required) it will segfault.
Unfortunately Azure Repos still uses an old ssh-rsa authentication so you need the following in your ~/.ssh/config (see: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops)