SSH SHA-256 key not accepted

[generalpax]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• [ x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ x] I've included steps to reproduce the bug.
• [x ] I've pasted the output of argocd version.

Describe the bug

I am trying to add a new repo to my argocd. Everytime i try i get the error "Unable to connect SSH repository: unknown error: ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type."

I also tried from cli and from gui and receiving the same error:

argocd repo add git@github.com:generalpax/ci-cd-istio-tekton.git --ssh-private-key-path ~/.ssh/id_rsa_github --insecure-skip-server-verification
FATA[0001] rpc error: code = Unknown desc = unknown error: ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type. 

Also regenerated my key with type rsa-sha2-256: ssh-keygen -t rsa-sha2-256

To Reproduce

• Install ArgoCD Operator on Openshift 4.8.15
• Create namespace and add argocd instance to it.
• Add repo to argo

Expected behavior

Connected github repo

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v2.1.2+7af9dfb.dirty
  BuildDate: 2021-09-02T21:28:55Z
  GitCommit: 7af9dfb3524c13e941ab604e36e49a617fe47d2e
  GitTreeState: dirty
  GoVersion: go1.17
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.0.0+f5119c0
  BuildDate: 2021-04-07T06:00:33Z
  GitCommit: f5119c06686399134b3f296d44445bcdbc778d42
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

Logs

[jannfis]

Are you able to use the same key with your local Git client to access Git Hub?

[jannfis]

I think this may be the reason: https://github.blog/2021-09-01-improving-git-protocol-security-github/

[Markieta]

We just went final brownout (January 11, 2022) and I am experiencing this same issue with SHA256 keys.

Does the argocd-repo-server SSH client not support SHA-2?

[Markieta]

Switching to Ed25519 as per GitHub's recommendation worked for me. You can update existing repositories this way.

ssh-keygen -t ed25519 -a 100 # Source: https://security.stackexchange.com/a/144044/29078

Add cat ~/.ssh/id_ed25519.pub as a new repo deploy key in GitHub/Bitbucket/etc.

cat ~/.ssh/id_ed25519 | base64 -w 0 && echo # copy output

kubectl get secrets -n argocd
kubectl edit -n argocd secret repo-<uid>

Replace sshPrivateKey: <encoded_private_key> with your new encoded private key.

[awknode]

Thanks for the info but this didn't work for me with argocd for some reason. IDK why all the sudden it won't accept the same keys I've been using all month.

[j7nw4r]

I just got bit by this as well. Was using an ssh-rsa. Our ArgoCD instance is new, so I don't have the experience of the previous brownout.

After reading the above article, Argo is probably using an older git client which does not support SHA2 sigs (or it is not configured).

As the above shows (and the article explains) moving away from ssh-rsa should fix this (ed25519 is "more secure", as of 01/11/22 anyway).

I also had to disconnect all the instances that I had directly configured (the one's that were not using a credential template)

Hope this helps.

[NiallJude]

This just hit us as well. No updates made to the cluster today. I would love to know what triggered it.

[luke-plausin-kidsloop]

We are impacted as of 15 minutes ago... Changes in Github security policy again?

EDIT: Generating a new ed25519 key as suggested above worked for us. Github have changed their instructions for generating SSH keys to use ed25519 by default.

ssh-keygen -t ed25519 -C "your_email@corp.com

I don't know why the old key was blocked as it was SHA-256 and 4096 bits, it seems like Github deprecated a lot of other things like cipher suites and MAC types so it might have been something more obscure.

[Markieta]

March 15, 2022 Changes made permanent. We’ll permanently stop accepting DSA keys. RSA keys uploaded after the cut-off point above will work only with SHA-2 signatures (but again, RSA keys uploaded before this date will continue to work with SHA-1). The deprecated MACs, ciphers, and unencrypted Git protocol will be permanently disabled.

[Kampe]

the docs literally still tell users to generate RSA keys: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

yet we cannot connect with keys generated the same way.

[j7nw4r]

The problem is that the git client that Argo uses doesn't do sha2 signatures. Use an ed25519 key.

[bpourian]

I know this issue is still open but just to say that we had a similar problem where argocd and github wouldnt sync because of the key generation and this did work for us: ssh-keygen -t ed25519 -a 100

[snooyen]

We were using Terraform tls_private_key resource to help automate creation of deploy keys on our ArgoCD repositories, but unfortunately ed25519 support is still under development.

In the meantime, we've reworked our Terraform module(s) to expect deploy keys to exist in our org's secrets store and regenerated them all using ssh-keygen -t ed25519 as described in previous comments.

[Silvenga]

To be clear, this is an ArgoCD problem (or rather the SSH client ArgoCD is using), SSH RSA keys aren't signed with SHA-1 (that would be nonsensical), the error relates to SHA-1 being used during the handshake. SSH client should being using SHA-256 (aka rsa-sha2-256) during handshakes.

https://datatracker.ietf.org/doc/html/rfc8332

All aspects of the "ssh-rsa" format are kept, including the encoded string "ssh-rsa". This allows existing RSA keys to be used with the new public key algorithms, without requiring re-encoding or affecting already trusted key fingerprints.

Being more clean, the format of the RSA key isn't the problem. Switching to Ed25519 just forces the client to not use ssh-rsa during handshakes (since handshakes should use ssh-ed25519, https://datatracker.ietf.org/doc/html/rfc8709).

[blakepettersson]

This issue might be related to golang/go#37278, specifically this comment. golang/crypto#211 might address this, but unknown when/if this will be merged...

[blakepettersson]

golang/crypto@6fad3dfc may have fixed this. Argo CD 2.7.x upgraded to golang.org/x/crypto v0.6.0 so might be worth trying out there?

[cooervo]

yeah this also started to happen on our side without any changes from us