ssh issues with encrypted identity files on MacOS

[dtrifiro]

SSH on MacOS provides the UseKeychain option (man ssh_config) that makes it possible to decrypt encrypted ssh identity files on-the-fly by retrieving the passphrase from MacOS' keychain.

This works with git clone and other git cli commands, but fails when using certain dvc operations (dvc exp list, dvc get, ...) because the ssh vendor we use for git operations and for remote operations (asyncssh) does not support Apple keychain integration.

Using UseKeychain can result in authentication issues (unexpected CloneError or Permission Denied errors).

The solution is to decrypt the ssh key and add it to the ssh agent:

ssh-add --apple-load-keychain ~/.ssh/ed255

(note that --apple-load-keychain used to be -A)

By adding the key to the ssh-agent, asyncssh is able to properly use the (decrypted) key.

First noted by @shcheklein in https://github.com/iterative/dvc/issues/7702#issuecomment-1304895509_

[pmrowla]

reposting from https://github.com/iterative/scmrepo/pull/210#issuecomment-1477225173

Porting the apple openssh to asyncssh is non-trivial and probably not what we want. What we should be doing in dulwich is checking for unsupported configuration options in our asyncssh vendor (like UseKeychain which specifies using the apple keychain behavior) and then fallback to the default dulwich SSH vendor instead (which just calls system ssh via subprocess, and would use apple's openssh build and work as expected).

We default to using asyncssh since it removes a dependency on ssh, which is not always available in windows (or in containers)

[efiop]

@pmrowla Thank you for clarifying! That sounds very good.

[pmrowla]

This issue probably should not have been moved into dvc-ssh, since it was a git+SSH problem (I'm assuming that at some point it got marked fs: ssh in the DVC repo).

However, the underlying issue w/asyncssh not supporting all OpenSSH flags does actually affect dvc-ssh, and the scmrepo change does not actually fix the problem in dvc-ssh, so I will keep this issue open.

Essentially, the macos UseKeychain option will now work properly for git operations (meaning clone/fetch on anything that uses DVC erepo), but it will still result in authentication errors if a DVC user is using an SSH DVC remote with a key stored in the macos/apple keychain.

We don't have a straightforward fallback mechanism for DVC remotes, since DVC was never using subprocess to run system ssh in the first place for DVC remotes (before asyncssh/sshfs we used paramiko, which also does not support UseKeychain without setting AddKeysToAgent)

To handle this properly for DVC remotes we would need an fsspec sftp implementation that is capable of using CLI ssh (fsspec's native ssh/sftp implementation uses paramiko)