Open bryannagle opened 3 years ago
Perhaps the most telling thing is that doing a third operation (with the freshly generated known hosts entry) fails host verification again.
Here's my analysis of this problem. You may be aware that if you've used git
and ssh
to clone/push/pull anything from gitlab.com
, it saves a host fingerprint to your known hosts file (typically ~/.ssh/known_hosts
). GitLab publicizes fingerprints for three different algorithms, which you can see all of with the ssh-keyscan
tool:
$ ssh-keyscan gitlab.com
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
Command-line git
and ssh
both use libssh
which uses OpenSSL and have support for all of these fingerprint methods. Julia's libgit2 uses libssh2
built against mbedTLS, which does not support all of these fingerprint methods — it only supports the classic ssh-rsa
fingerprints. That mismatch is the root of the problem.
The specific issue is that when you use git
or ssh
to clone/push/pull from GitLab, it negotiates one fingerprinting method, and it tends to pick the newer, more secure algorithms; specifically ecdsa-sha2-nistp256
is preferred, so that's what's recorded in most people's known hosts files. That would be fine if our libgit2/libssh2/mbedtls understood that method, but it doesn't, so it concludes when trying to verify the identity of gitlab.com
that the host identity is invalid since it can't verify the fingerprint. If you had never connected to gitlab.com
before, you'd actually get a much better error message telling you to run ssh-keyscan gitlab.com
, which would fix the issue since you'd get all of the fingerprints, including the one that libgit2/libssh2/mbedtls understands.
Why was this not a problem before? Because libgit2 defaults to not verifying host identities at all (😬). This is bad, so in 1.6 we turned on host identity verification by default. You can, however, opt out of SSH host identity verification by setting the shell environment variable JULIA_SSH_NO_VERIFY_HOSTS=gitlab.com
(or, if you just want to not verify anything, use **
as the value). See https://github.com/JuliaLang/NetworkOptions.jl for all the possibilities.
Why isn't this a problem for GitHub? Because they only advertise one kind of SSH fingerprint and it's ssh-rsa
:
$ ssh-keyscan github.com
# github.com:22 SSH-2.0-babeld-0913e7e1
# github.com:22 SSH-2.0-babeld-0913e7e1
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
# github.com:22 SSH-2.0-babeld-0913e7e1
So our libgit2/libssh2/mbedtls stack understands that known hosts entry and everything works fine.
For the user, the short-term secure fix is to make sure that the known hosts file has an ssh-rsa
fingerprint for gitlab.com
, i.e. this line:
gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
You also want to make sure this is the only fingerprint for gitlab.com
in the file, which is a bit unfortunate, but otherwise you'll still have issues because of the following. When you use git
and ssh
they will choose the newer methods if those fingerprints are present, and then they will also record entries for specific host IP addresses in addition to the entries for the gitlab.com
hostname, but only for the fingerprint method that's chosen. If a newer fingerprint type like ecdsa-sha2-nistp256
exists in known hosts for gitlab.com
then git
or ssh
will use that method and record a corresponding ecdsa-sha2-nistp256
fingerprint for the server IP address. Later, if Julia's libgit2 tries to connect to that same IP address, which is likely, it will fail because it doesn't know how to check that kind of fingerprint.
Longer term, the solution is to make sure that our libgit2/libssh2 stack knows how to compute these other kinds of fingerprints. One solution would be to switch from mbedTLS to OpenSSL. Another would be to add support for the other fingerprint methods with mbedTLS. @staticfloat has opened an issue with the libssh2 project for this.
For those wondering about the state of libssh2, there hasn't been a release in over two years, even to mitigate active CVEs. The maintainers have not responded to requests for a new release since April 1.
Yeah, the libgit2+libssh2 situation is a bit of a trainwreck. Libgit2 has many problems itself (insecure defaults all over the place, inability to do many things that work fine in CLI git); it also depends on libssh2, which, as noted is no longer actively maintained and has unfixed CVEs. My conclusion is that we have to drop both.
Do you think the best way is to go back to git
CLI?
Yes, I'm in the middle of writing up an issue on Pkg.jl
This may be able to be improved now that libssh2 1.10 has been tagged with ECDSA support for mbedTLS. Not sure if a corresponding libgit2 release needs to be made or we can just update libssh2 directly.
I got this when I first tried to update the registry for 1.9 (copied over my v1.8 Project.toml
file, omitted the Manifest.toml
and hoping to let it re-resolve):
(@v1.9) pkg> up
Updating registry at `~/.julia/registries/General`
Updating git-repo `git@github.com:JuliaRegistries/General.git`
SSH host verification: the identity of the server `github.com:22` does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file.
Updating registry at `~/.julia/registries/HolyLabRegistry`
Updating git-repo `git@github.com:HolyLab/HolyLabRegistry.git`
SSH host verification: the identity of the server `github.com:22` does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file.
┌ Error: Some registries failed to update:
│ — /home/tim/.julia/registries/General — failed to fetch from repo: failed to fetch from git@github.com:JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)
│ — /home/tim/.julia/registries/HolyLabRegistry — failed to fetch from repo: failed to fetch from git@github.com:HolyLab/HolyLabRegistry.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)
└ @ Pkg.Registry ~/src/julia-master/usr/share/julia/stdlib/v1.9/Pkg/src/Registry/Registry.jl:449
ERROR: expected package `RegistryCompatTools [0f4576a4]` to be registered
This happened perhaps because I had a few dev
ved unregistered packages (e.g., RegistryCompatTools
above). Removing the ones I didn't want and manually dev
ving the ones I did fixed the issue, but the warning is obviously very misleading.
Can't you use JULIA_PKG_USE_CLI_GIT="true"
to avoid these issue?
Is this still an issue?
When using Julia 1.6, running "activate" on a module that has unregistered dependencies that use git ssh urls to private repositories, the update will fail if an entry for that repository host (i.e. gitlab) is present in the users ~/.ssh/known_hosts file. Removing that entry fixes the problem, but then the entry is re-added when the user pushes to git.
Steps to Reproduce:
Repl Example:
(LiquidIAM) pkg> update Updating registry at
~/.julia/registries/General
Updating git-repogit@gitlab.com:LiquidAnalytics/opensource/liquiddatautils.jl.git
SSH host verification: the identity of the servergitlab.com:22
does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file. ERROR: failed to fetch from git@gitlab.com:LiquidAnalytics/opensource/liquiddatautils.jl.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)(LiquidIAM) pkg>
shell> rm ~/.ssh/known_hosts
(LiquidIAM) pkg> update Updating registry at
~/.julia/registries/General
Updating git-repogit@gitlab.com:LiquidAnalytics/opensource/liquiddatautils.jl.git
Updating git-repogit@gitlab.com:LiquidAnalytics/opensource/dgraph.jl.git
Updating git-repogit@gitlab.com:LiquidAnalytics/LiquidDecisions/Goals/DistributeGoal.git
Installed AWS ─ v1.33.1 No Changes to~/Workspace/liquid/liquidiam.jl/Project.toml
Updating~/Workspace/liquid/liquidiam.jl/Manifest.toml
[fbe9abb3] ↑ AWS v1.32.0 ⇒ v1.33.1 Progress [========================================>] 6/6 6 dependencies successfully precompiled in 22 seconds (68 already precompiled)(LiquidIAM) pkg>