Cannot dev private repos: libgit2 uses protocol phased out by Github today? (SHA-1)

[grahamas]

Dev'ing my private repos is failing today, where it worked yesterday. git pull works fine. I'm on julia v1.6.2 Ubuntu and v1.6.5 Windows.

As the final phase in updating their SSH security protocols, today GitHub stopped accepting RSA keys with SHA-1.

Per GitHub:

March 15, 2022  

Changes made permanent.

We’ll permanently stop accepting DSA keys. RSA keys uploaded after the cut-off point above will work only with SHA-2 signatures (but again, RSA keys uploaded before this date will continue to work with SHA-1). The deprecated MACs, ciphers, and unencrypted Git protocol will be permanently disabled.

It seems that libgit2 uses SHA-1, so I think it's libgit2's fault. At least, yesterday dev'ing worked, and today git pull still works. I'm a little confused because my RSA key was uploaded before today, but I can't argue with the error:

ERROR: failed to clone from git@github.com:[PRIVATE_REPO], error: GitError(Code:EEOF, Class:SSH, ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

In theory, you could use a new key type, but libgit2 only notices RSA or something weird like that (see https://github.com/JuliaLang/Pkg.jl/issues/911, I think).

Where does the recent PR https://github.com/JuliaLang/julia/pull/43250 stand julia release-wise? Is there likely a workaround for older julia versions? (I'm on a cluster and beholden to admins for updating).

Related to https://github.com/JuliaLang/Pkg.jl/issues/2679

[KristofferC]

Don't you just have to generate a new key as described in https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent?

[grahamas]

Don't you just have to generate a new key as described in https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent?

No, the issue is not with the RSA key itself, but with the signing algorithm employed by libgit2. My RSA key works fine through command line git.

Just to double check, I followed the instructions in the link to generate both an RSA and an ED25519. The RSA failed as above, while the ED25519 could not be found by libgit2 (as in previous issues)

[KristofferC]

From https://github.blog/2021-09-01-improving-git-protocol-security-github/

If you’re using libgit2 or another piece of code using libssh2, we recommend you use libssh2 1.9.0 or newer and an ECDSA key, since it does not yet support RSA with SHA-2.

Julia 1.6 uses libssh 1.9.1 so it should work with ECDSA?

[grahamas]

I hadn't tried ECDSA before (I did now), but it runs into the same problem as using ED25519: libgit2 fails to automatically detect the private key file, prompts for the location, and then continues to prompt for the location twice more before erroring out due to "maximum number of prompts reached."

From all the issues I've read, libgit2 in Julia only can handle RSA keys generated with PEM. (https://github.com/JuliaLang/Pkg.jl/issues/911#issuecomment-640399940; https://github.com/JuliaLang/Pkg.jl/issues/1516#issuecomment-654581794)

[KristofferC]

And you really need to use ssh, you cannot use https for the git remote?

[DilumAluthge]

I ran into this recently, and I was able to fix it simply by doing export JULIA_PKG_USE_CLI_GIT=true. Can you try and seee if that works for you?

[KristofferC]

That requires Julia 1.7.

[joaquimg]

And you really need to use ssh, you cannot use https for the git remote?

https is very painful as it requires user and password before every clone/update of private repos

[KristofferC]

So I just tried deving a private package on GitHub using Julia 1.6 with https and it didn't ask for any user/password and it worked fine so I guess that means it is cached?

[joaquimg]

Two issues

• with 2FA I needed tokens instead of passwords https://docs.github.com/pt/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
• my user/token don't get memorized so it is painful... I am on windows (don't if this is the reason)

[grahamas]

Apparently you need to run git config credential.helper store and then it will store your credentials in plain text next time you enter them. I really don't like this solution because of the security concerns, but there is a perfectly fine solution in v1.7 (export JULIA_PKG_USE_CLI_GIT=true), so I'm ok considering this workaround one of those unfortunate realities of using an older version, even if it is LTS.

So, to recap, the workaround is either: 1) Upgrade to v1.7+ and use export JULIA_PKG_USE_CLI_GIT=true, or 2) Switch all private repos and registries to HTTPS, and use git config credential.helper store

I'll close this now, but if there's any way to get that env flag into the next release of v1.6, that would be way better. I would argue it's a bugfix.

[joaquimg]

Backporting the pkg manager would be amazing in a julia 1.6 patch, since it is LTS.

[joaquimg]

@grahamas I could do option 2 in windows, but not in linux. Which system did you use?

[grahamas]

Oh no, I use both but I only tested it on Windows; my Linux system is the one that I can't control the Julia version on, too :/

I'll poke around a bit more and report back here if I find anything. If we can't find a way to cache username and password on Linux, I'll re-open, because re-entering tokens/passwords for every pull or update is unfeasible.

[joaquimg]

It worked on linux by fixing my .giconfig that was pointing to a netrc file. Now it is back to default and points to .git-credentials. For completeness, you can modify directly the .git-credentials file adding something like the line: https://MYUSER:MYTOKEN@github.com

[sairus7]

Lost several hours with this... But found a surspising third solution, as they wrote in a blog-post:

Keys with a valid_after date before the deadline (November 2, 2021) may continue to use SHA-1 signatures for the time being.

So, look for an old ssh key (maybe on some other machine?), that was generated before November 2021 (If you're lucky enough).

I think github wanted to make a smoooth transition, which instead resulted in a kind of a time bomb, when keys generated after November 2021 - suddenly stopped working at March 2022.

[grahamas]

I strongly suspect that there was a deprecation warning that libgit2 didn't propagate...

[GunnarFarneback]

For those coming here later looking for information:

From https://github.blog/2021-09-01-improving-git-protocol-security-github/

If you’re using libgit2 or another piece of code using libssh2, we recommend you use libssh2 1.9.0 or newer and an ECDSA key, since it does not yet support RSA with SHA-2.

Julia 1.6 uses libssh 1.9.1 so it should work with ECDSA?

The ECDSA support is conditional on using openssl as crypto backend in libssh2. Julia is built with mbedtls as crypto backend. To find out what key types are supported for different crypto backends, search for LIBSSH2_ in the libssh2 sources (for the appropriate release). Specifically in the files src/libgcrypt.h, src/mbedtls.h, src/openssl.h, src/wincng.h.

Also see https://discourse.julialang.org/t/local-registry-rsa-key-problem/78188/4 for more information.

[KristofferC]

Also, I just want to say that we backported the JULIA_PKG_USE_CLI_GIT to the next LTS (1.6.6) which should be out in a few days.

[KristofferC]

I think we can keep this open for some extra visibility.

[staticfloat]

Looking into this a bit, without any changes to Julia, you can fix the issue by using a different SSH key: you can just generate a new ECDSA key (e.g. those generated by ssh-keygen -t ecdsa) instead of an RSA key and use that instead. Julia v1.8+ contains support to natively read ECDSA keys. If you're using ssh-agent to store your keys, the agent can actually help out and that works all the way back to v1.4 (that's just the earliest version I tested, it may work even farther back).

To get RSA keys working again, we need libssh2 to implement SHA-256 signing of RSA keys with MbedTLS as the backing crypto library. I opened an issue to ask them to fix it for us, but if they dawdle, it's likely something someone with some C experience can hack together in an afternoon. It's basically hooking up the pipes from libssh2 to MbedTLS, and ensuring that you don't break SHA-1 while you're at it.

[GunnarFarneback]

Julia v1.8+ contains support to natively read ECDSA keys.

That's great. Where is it implemented?

[staticfloat]

It's inside of libssh2 ever since version 1.10.0; it's not in Julia code, it's just the native libgit2->libssh2->mbedtls stack.

[GunnarFarneback]

I guess I was confused when I thought the mbedtls support for ECDSA keys entered libssh2 after the 1.10.0 release and only was available in master. Now it would be nice if the ECDSA key could be found automatically without needing an SSH_KEY_PATH hint.

[sairus7]

So, since https://github.com/JuliaLang/julia/pull/44767 is merged, why do I keep running into this and manually provide id_ecdsa path?

    Updating registry at `C:\Users\sairu\.julia\registries\JuliaRegistry`
    Updating git-repo `git@github.com:MyPrivateOrg/JuliaRegistry.git`
Private key location for 'git@github.com' [C:\Users\sairu\.ssh\id_rsa]: C:\Users\sairu\.ssh\id_ecdsa
    Updating registry at `C:\Users\sairu\.julia\registries\General.toml`