search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
16.23k stars 4.89k forks source link

ArgoCD Repo Server stops pulling git repositories due to Azure Devops Repos current sunset SSH-RSA strategy #17634

Closed deB4SH closed 4 days ago

deB4SH commented 1 month ago

Hi all, Microsoft published a blogpost Feb 15th 2024 to sunset the ssh-rsa support and wants to migrate to rsa-sha2-256/512. Blog Post: https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/

Based on their schedule everyone that still uses their service should be in Phase 2 where an throttling/delay is in place and an error is shown with following message.

“ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

This error is also shown within argo. grafik

After searching a bit - it seems like that golang/crypto already support rsa-sha2-256/512 but sadly starting from v0.21.0. (https://github.com/golang/crypto/commit/6fad3dfc)

Argo seems to use v0.19.0 https://github.com/argoproj/argo-cd/blob/1bddee2e5dfff35613847eef9a2c0e6818976dc3/go.mod#L85

Also found a relevant issue in this regard: https://github.com/argoproj/argo-cd/issues/7600

Checklist:

Describe the bug Argo is currently unable to pull git repositories provided by azure devops repos and stops after receives the delay error.

To Reproduce Pull a repository from their service with a ssh key.

Expected behavior Argo pulls changes or state from git repository.

Version

argocd@argocd-core-server-588df95858-5jcc7:~$ argocd version
argocd: v2.9.3+6eba5be
  BuildDate: 2023-12-01T23:05:50Z
  GitCommit: 6eba5be864b7e031871ed7698f5233336dfe75c7
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64

Logs

Logs from repo-server:

Stream closed EOF for argocd/argocd-core-repo-server-6b48b5bc4b-t5xfj (copyutil)
repo-server time="2024-03-27T06:40:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2024-03-27T06:40:28Z" grpc.time_ms=0.019 span.kind=server system=grpc
repo-server time="2024-03-27T06:40:34Z" level=error msg="finished unary call with code Unknown" error="unknown error: remote: Command git-upload-pack: ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://aka.ms/ado-ssh-rsa-deprecation." grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-03-27T06:40:29Z" grpc.time_ms=5616.032 span.kind=server system=grpc
maxkaemmerer commented 4 weeks ago

We are having the same issue and currently can't sync our applications anymore.

Any progress on this?

mbu147 commented 4 weeks ago

We are having the same issue and currently can't sync our applications anymore.

Any progress on this?

+1

{
    "Version": "v2.10.4+f5d63a5",
    "BuildDate": "2024-03-18T08:09:23Z",
    "GitCommit": "f5d63a5c77d2e804e51ef94bee3db441e0789d00",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.3+gf03cc04",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}
crenshaw-dev commented 3 weeks ago

My understanding is that this isn't an issue with golang's crypto library. Rather it has to do with the hash algorithms ssh accepts in Ubuntu 22.04 (the version Argo CD images are built on).

It looks like Ubuntu 22.04 does support the listed algorithms (based on this guide), but maybe neither algorithm is enabled by default in the handshake. You might have to explicitly enable that host key algorithm.

$ ssh -Q sig 
ssh-ed25519
sk-ssh-ed25519@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
maartengo commented 3 weeks ago

I'm not too familiar with all the internal workings of SSH, but would it be a solution to change the known_hosts entry for azure devops to: ssh.dev.azure.com rsa-sha2-512 instead of ssh.dev.azure.com ssh-rsa?

I've changed this within my setup using configs.ssh.knownHosts. I'm not sure if this helps during the brownout, but it seems to be working at the moment.

maartengo commented 3 weeks ago

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

siawkz commented 2 weeks ago

argocd repo add with --insecure-skip-server-verification flag will bypass the verification which causes the issue.

maartengo commented 2 weeks ago

My previous 'workaround' doesn't work, sadly. Is /home/argocd/.ssh/config not the right place to put the ssh config?

Disabling the server verification doesn't sit well with me either, so I hope there is some other solution.

siawkz commented 2 weeks ago

My previous 'workaround' doesn't work, sadly. Is /home/argocd/.ssh/config not the right place to put the ssh config?

Disabling the server verification doesn't sit well with me either, so I hope there is some other solution.

Do you encounter issue pulling git repositories directly on the host machine?

maartengo commented 2 weeks ago

Do you encounter issue pulling git repositories directly on the host machine?

I haven't tested this, but I can at least say that the config is used with a regular ssh ssh.dev.azure.com call; if I input invalid config (e.g. foobar as HostkeyAlgorithms) then the ssh command fails.

siawkz commented 2 weeks ago

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

  1. Generate new public private key ssh-keygen -t rsa-sha2-512
  2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

zamedic commented 2 weeks ago

I can confirm what is being mentioned here. During the microsoft brownouts, we are able to work locally with Azure Devops GIT using SSH keys, however all of our argo instances fail to connect citing the error mentioned above.

AleksuKey commented 2 weeks ago

We just got into this issue right now. We did the following steps to stop having the "You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally." error:

ssh-keygen -f key_file_name -t rsa-sha2-512

ssh.dev.azure.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
vs-ssh.visualstudio.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

Hope this helps as a temporal workaround.

maxkaemmerer commented 2 weeks ago

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

rouke-broersma commented 2 weeks ago

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.

@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

  1. Generate new public private key ssh-keygen -t rsa-sha2-512
  2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

maxkaemmerer commented 2 weeks ago

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.

@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

  1. Generate new public private key ssh-keygen -t rsa-sha2-512
  2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

bartoszpyrek commented 2 weeks ago

On the ArgoCD 2.8.4 version rsa-sha2-256 is working, but the important thing to check - is in the k8s secret with sshPrivateKey for Azure DevOps if appropriately put the new lines, because when we put this private key in the Azure Key Vault as a secret then "-----END OPENSSH PRIVATE KEY-----" part was not in the new line but it was as a part of the key. After fixing this, ArgoCD can sync properly with Azure DevOps repos.

patst commented 2 weeks ago

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

We have this config since a few weeks ago. Nevertheless, we observed the clone errors today.

I would expect you will see the errors appearing again during the next brownout period. :-/

rouke-broersma commented 2 weeks ago

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

I've been looking through the code, it looks to me like the /home/argocd/.ssh/config is not being used, because a different SSH config dir is being set: https://github.com/argoproj/argo-cd/blob/4d527f87056275053ca584d6ef2732ad57e5fa08/common/common.go#L75

This makes sense, because this path is also where the argocd-ssh-known-hosts-cm is mounted by the argocd helm chart:

- mountPath: /app/config/ssh
     name: ssh-known-hosts

So it seems to me like we should extend the argocd-ssh-known-hosts-cm with the config file, instead of mounting a new configmap, so the config file is placed in the actual used ssh config folder.

AleksuKey commented 2 weeks ago

@rouke-broersma About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors) We are also very hesitant to add the --insecure flag. This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found. We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

Probably the same happened to us and that's why we saw it solved...

maxkaemmerer commented 2 weeks ago

@rouke-broersma About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors) We are also very hesitant to add the --insecure flag. This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found. We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

I've been looking through the code, it looks to me like the /home/argocd/.ssh/config is not being used, because a different SSH config dir is being set:

https://github.com/argoproj/argo-cd/blob/4d527f87056275053ca584d6ef2732ad57e5fa08/common/common.go#L75

This makes sense, because this path is also where the argocd-ssh-known-hosts-cm is mounted by the argocd helm chart:

- mountPath: /app/config/ssh
     name: ssh-known-hosts

So it seems to me like we should extend the argocd-ssh-known-hosts-cm with the config file, instead of mounting a new configmap, so the config file is placed in the actual used ssh config folder.

We had a syntax error in the ssh-config mounted at /home/argocd/.ssh/config and that error showed up in the Argo UI, so I'd assume it is being used.

Very possible that we tested after the brownout however.

FlorisFeddema commented 2 weeks ago

I did some testing yesterday and believe the issue might be in Argo.

It seems like the git implementation used in argo does some parts with the Go user agent and some with the OpenSSH user agent. Both processes probably need to use the rsa-sha2-256/512 signature to be compliant with Azure DevOps. It might also explain why changing git config does not solve the issue, maybe not both processes use the same git config location. (maybe if both /home/argocd/.ssh/config and /app/config/ssh are updated it might work?)

To check which of the 2 processes is using the wrong signature I used the reposerver container to manually do a git fetch (uses OpenSSH user agent) with extra verbosity. (GIT_SSH_COMMAND="ssh -vvv -i ./../key -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts" git fetch).

This results in the following (removed a lot of content that is not ):

...
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256
...
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: KEX algorithms: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512
debug2: ciphers ctos: aes256-cbc,aes192-cbc,aes128-cbc,aes128-ctr,aes256-ctr
debug2: ciphers stoc: aes256-cbc,aes192-cbc,aes128-cbc,aes128-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: Server host key: ssh-rsa SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og
debug3: record_hostkey: found key type RSA in file /app/config/ssh/ssh_known_hosts:13
debug3: load_hostkeys_file: loaded 1 keys from ssh.dev.azure.com
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:13
debug3: load_hostkeys_file: loaded 1 keys from ssh.dev.azure.com
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.dev.azure.com' is known and matches the RSA host key.
debug1: Found key in /app/config/ssh/ssh_known_hosts:13
debug1: Will attempt key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: Next authentication method: publickey
debug1: Offering public key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo
Authenticated to ssh.dev.azure.com ([40.74.28.31]:22) using "publickey".
...

It shows that it uses rsa-sha2-512, and thus should work with Azure DevOps during the brownouts and after the removal of ssh-rsa, but since we have not seen any other brownouts I could not run this during brownout. Since openSSH uses the correct signature, it probably is the go-git client that does not use the correct signature thus breaking ArgoCD with Azure DevOps.

While for Github the issue was to use an elliptic key, this should be fixed in argo or the go-git client. It also looks like Flux users also had problems with host key algoritms. They added a flag for this to the project (https://github.com/fluxcd/source-controller/pull/711).

maxkaemmerer commented 1 week ago

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us. @siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

  1. Generate new public private key ssh-keygen -t rsa-sha2-512
  2. Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

We are having issues again today, so it seems like this did not infact resolve

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

We are having issues again today so it seems like it just worked for us because the brownout had ended

FlorisFeddema commented 1 week ago

Looks like the brownout is over again...

danijam commented 1 week ago

Joining this thread as we also just recently regenerated our SSH key as Microsoft advised but then during the next brownout observed the issue. Reading through this I'm not certain we've identified a solution or if we need one (aka once ADO kill ssh-rsa it just works)...

Anyone considered using HTTPS? Any pros or cons I should be aware of over SSH? I'm guess SSH security is considered better asymmetric crypto versus a PAT based http basic auth? Anything else?

patst commented 1 week ago

https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/

According to their schedule the brownout times and lengths will increase to 8 and 12 hours per day in the next week(s). That will be very annoying to sit out and hope it will take the correct algorightm afterwards

I just took a look at the code and have no idea how everything is wired together, but these parts look interesting:

https://github.com/argoproj/argo-cd/blob/master/util/git/client.go#L267-L279 where the git ssh client is constructed

I took a look in our reposerver with cat /app/config/ssh/ssh_known_hosts and the result is:

ssh.dev.azure.com rsa-sha2-512 ....

which is what we defined as custom config.

Nevertheless, we are getting the errors. Maybe we need a way to configure custom algorithms here? https://github.com/argoproj/argo-cd/blob/master/util/git/client.go#L267

rouke-broersma commented 1 week ago

@patst we are coming to the same conclusions by digging through the code and we hope to test this to confirm and hopefully submit a patch.

aurel4oxand commented 1 week ago

On the ArgoCD 2.8.4 version rsa-sha2-256 is working, but the important thing to check - is in the k8s secret with sshPrivateKey for Azure DevOps if appropriately put the new lines, because when we put this private key in the Azure Key Vault as a secret then "-----END OPENSSH PRIVATE KEY-----" part was not in the new line but it was as a part of the key. After fixing this, ArgoCD can sync properly with Azure DevOps repos.

Hi, I'm using ArgoCD v2.7.11

Thanks to @bartoszpyrek I managed to get this working by :

Public key is of form : 

ssh-rsa AAAAB3NzaC1y..... user@host

image

Note the new line in yellow after -----END OPENSSH PRIVATE KEY-----

This worked fine 👌 image

Also, it seems there's no need to update know-hosts configuration, nor skip server verification

Finally, I updated my secret which contains my SSH private key with the new one, also with the newline at the end, and restarted argocd-repo-server pods

I manually deleted repositories which uses the old SSH key and let ArgoCD re-create them (ArgoCD manages itself its configuration, using App of App pattern)

Now everything is fine, let's see if this works during next days...

Good luck 🤞

rouke-broersma commented 1 week ago

@aurel4oxand let me guess you tested this somewhere in the last 40 minutes? The brownout is over :)

aurel4oxand commented 1 week ago

@aurel4oxand let me guess you tested this somewhere in the last 40 minutes? The brownout is over :)

Damned, you're right.. 😭 completely forgot about the brown out phase..

rouke-broersma commented 1 week ago

As far as we can tell (we are still trying to confirm but it is EOD) the issue is here:

https://github.com/argoproj/argo-cd/blob/0f11dfb5961361807962aafc68b11426b8a47490/util/git/ssh.go#L54

The custom ssh client config overwrites the default ssh client that go-git normally uses:

https://github.com/go-git/go-git/blob/1ddd78a55f803c9ed0fac6a56d201271fef21eeb/plumbing/transport/client/client.go#L19

The default ssh client config reads the ssh config file and known hosts file from the current user (this would be ~/.ssh/config and ~/.ssh/known_hosts) to construct the client.

See:

https://github.com/go-git/go-git/blob/1ddd78a55f803c9ed0fac6a56d201271fef21eeb/plumbing/transport/ssh/common.go#L26

However it does not do this if the auth method contains a custom client config (as argocd creates). In that case it uses the host key algorithms as set in the client config, which in our case is nil because argocd does not set them in the client config.

In that case go ssh uses 'sane defaults', see:

https://pkg.go.dev/golang.org/x/crypto/ssh#ClientConfig

Any string returned from a PublicKey.Type method may be used, or any of the CertAlgo and KeyAlgo constants.

Which are the following:

https://pkg.go.dev/golang.org/x/crypto/ssh#pkg-constants

const (
    CertAlgoRSAv01        = "ssh-rsa-cert-v01@openssh.com"
    CertAlgoDSAv01        = "ssh-dss-cert-v01@openssh.com"
    CertAlgoECDSA256v01   = "ecdsa-sha2-nistp256-cert-v01@openssh.com"
    CertAlgoECDSA384v01   = "ecdsa-sha2-nistp384-cert-v01@openssh.com"
    CertAlgoECDSA521v01   = "ecdsa-sha2-nistp521-cert-v01@openssh.com"
    CertAlgoSKECDSA256v01 = "sk-ecdsa-sha2-nistp256-cert-v01@openssh.com"
    CertAlgoED25519v01    = "ssh-ed25519-cert-v01@openssh.com"
    CertAlgoSKED25519v01  = "sk-ssh-ed25519-cert-v01@openssh.com"

    // CertAlgoRSASHA256v01 and CertAlgoRSASHA512v01 can't appear as a
    // Certificate.Type (or PublicKey.Type), but only in
    // ClientConfig.HostKeyAlgorithms.
    CertAlgoRSASHA256v01 = "rsa-sha2-256-cert-v01@openssh.com"
    CertAlgoRSASHA512v01 = "rsa-sha2-512-cert-v01@openssh.com"
)

Where ssh-rsa is listed first, and a likely deduction is that since it's listed first it might be tried first. Since azure devops does not deny ssh-rsa until after the handshake has already been performed (because they want to send back an error message, they first have to accept the connection, to then close it with an error) argocd/go-git/go-ssh decide to use ssh-rsa as the preferred host key algorithm.

Now from the go-ssh docs you might infer that you can set rsa-sha2 as the keytype to use in the known_hosts file, except:

// KeyAlgoRSASHA256 and KeyAlgoRSASHA512 are only public key algorithms, not // public key formats, so they can't appear as a PublicKey.Type. The // corresponding PublicKey.Type is KeyAlgoRSA. See RFC 8332, Section 2. KeyAlgoRSASHA256 = "rsa-sha2-256" KeyAlgoRSASHA512 = "rsa-sha2-512"

So this is a no-go. So the only fix, I think, would be to modify https://github.com/argoproj/argo-cd/blob/0f11dfb5961361807962aafc68b11426b8a47490/util/git/ssh.go#L54 to either have a hardcoded list of supported algorithms or to read the supported algorithms from ~/.ssh/config.

thiagoanegreiros commented 1 week ago

Right now it is giving me this error

vinicius-bastos commented 1 week ago

I'm having the same issue today :/

evanmcneal commented 1 week ago

We are experiencing this as well and obviously Microsoft Azure DevOps only supports rsa as outlined here: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops

We are very worried about this impending deadline of May 1st as we have 15 or so clusters pulling repos from Azure DevOps and obviously do not want to have downtime where services can't be updated.

Hopefully we can get a fix soon and have enough time to verify it!

xenochor commented 1 week ago

I've yet to do further confirmation, but I do believe the underlying problem is related to an old work-around and Azure DevOps' failure to move beyond rsa to ecdsa or ed25519.

The work-around in question is at https://github.com/go-git/go-git/blob/1ddd78a55f803c9ed0fac6a56d201271fef21eeb/plumbing/transport/ssh/common.go#L140. It results in a host key algorithm of ssh-rsa regardless of the reasonable system defaults since the published host key is of type rsa and rsa-sha2-256 or rsa-sha2-512 are not considered public key formats. (BTW, hacking known hosts won't help as that field is ignored when parsing the file)

A few options: 1) Remove that work-around entirely; 2) Work around the work-around by changing an "ssh-rsa" entry to "rsa-sha2-512" or expanding it to ["rsa-sha2-512", "rsa-sha2-256", "ssh-rsa"]; or, 3) Extend PublicKeysWithOptions to implement git.AuthMethod and supply a client config that overrides the HostKeyAlgorithms.

Vinaum8 commented 1 week ago

We just got into this issue right now. We did the following steps to stop having the "You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally." error:

* Created a new pair of SSH keys with SHA2 512 with:

ssh-keygen -f key_file_name -t rsa-sha2-512

* Replaced the keys in ArgoCD and Azure DevOps.

* Changed known hosts values in ArgoCD chart values. The value is _configs.ssh.knownHosts_. We replaced the default Azure DevOps known hosts with the following ones:
ssh.dev.azure.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
vs-ssh.visualstudio.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

Hope this helps as a temporal workaround.

i tried here and working! i'll to follow this in the week and next week. Thanks for tip @AleksuKey

xenochor commented 1 week ago

We just got into this issue right now. We did the following steps to stop having the "You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally." error:

* Created a new pair of SSH keys with SHA2 512 with:

ssh-keygen -f key_file_name -t rsa-sha2-512

* Replaced the keys in ArgoCD and Azure DevOps.

* Changed known hosts values in ArgoCD chart values. The value is _configs.ssh.knownHosts_. We replaced the default Azure DevOps known hosts with the following ones:
ssh.dev.azure.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
vs-ssh.visualstudio.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

Hope this helps as a temporal workaround.

i tried here and working! i'll to follow this in the week and next week. Thanks for tip @AleksuKey

Based on https://github.com/golang/crypto/blob/0da2a6a1bbc8e689a335bea68b5cc0e3e8728854/ssh/knownhosts/knownhosts.go#L190, I really don't think the modified known hosts does anything at all!

rouke-broersma commented 1 week ago

@xenochor

There's also this:

// KeyAlgoRSASHA256 and KeyAlgoRSASHA512 are only public key algorithms, not // public key formats, so they can't appear as a PublicKey.Type. The // corresponding PublicKey.Type is KeyAlgoRSA. See RFC 8332, Section 2. KeyAlgoRSASHA256 = "rsa-sha2-256" KeyAlgoRSASHA512 = "rsa-sha2-512"

So even if the key type from known hosts was used I don't think rsa-sha2 would be valid.

pimjansen commented 1 week ago

Also hitting this and it gets pretty urgent. Who is working on this?

rouke-broersma commented 1 week ago

Possible implementations for a fix:

  1. Set a sane list of hardcoded defaults for HostKeyAlgorithms as is already done for KexAlgorithms, this would probably follow the crypto/ssh defaults except ssh-rsa, ssh-dss (since they are considered insecure): https://pkg.go.dev/golang.org/x/crypto/ssh#pkg-constants
    1. This may break users that don't use official container images that are using a set of host key algorithms that are not in the default images, or not in the crypto/shh list of defaults, but should work for official images
  2. Use ssh_config.DefaultUserSettings to read from ~/.ssh/config so users can supply a list of HostKeyAlgorithms using a configmap: https://github.com/go-git/go-git/blob/1ddd78a55f803c9ed0fac6a56d201271fef21eeb/plumbing/transport/ssh/common.go#L26
    1. This requires user input for all affected users but seems to be doable
  3. Use skeema/knownhosts in argocd/util/git/ssh but modify the HostKeyAlgorithms by removing ssh-rsa and adding rsa-sha2
    1. This method reads the allowed HostKeyAlgorithms from the known_hosts file. This works perfectly for ecdsa and the like but unfortunately this method as implemented by go-git simply does not work for rsa-sha2 because they key type for both ssh-rsa and rsa-sha2 is ssh-rsa , this can be read in https://pkg.go.dev/golang.org/x/crypto/ssh#pkg-constants
      1. // CertAlgoRSASHA256v01 and CertAlgoRSASHA512v01 can't appear as a Certificate.Type (or PublicKey.Type), but only in ClientConfig.HostKeyAlgorithms.
    2. This method is already the method being used by go-git (which argocd relies on in this case), see: https://github.com/go-git/go-git/blob/1ddd78a55f803c9ed0fac6a56d201271fef21eeb/plumbing/transport/ssh/common.go#L140
      1. len(config.HostKeyAlgorithms) == 0 because the custom ClientConfig used by ArgoCD does not set the HostKeyAlgorithms, so the known_hosts file is used, which sets the allowed algorithm to ssh-rsa because that is they key type for both ssh-rsa and rsa-sha2
Gottox commented 1 week ago

What helped us is to get into contact with the Microsoft support, mentioning this issue. This is a bad workaround and should be solved in ArgoCD/go-git, but as long as there's no workaround in place, the MS support is the only party that can help here.

danijam commented 1 week ago

@Gottox what were Microsoft able todo for you? Turn off the brown out?

karol-pawlowski commented 1 week ago

Could you tell, did any of the above mentioned workarounds helped you ? (many of the answers seem to be verified outside of brownout period). The brownout periods start to hit our organization development teams more and more. Is there an action taken by the argo team ?

rouke-broersma commented 1 week ago

Could you tell, did any of the above mentioned workarounds helped you ? (many of the answers seem to be verified outside of brownout period). The brownout periods start to hit the our organization development teams more and more. Is there an action taken by the argo team ?

There are two workarounds:

  1. Migrate from SSH to PAT
  2. Disable TLS verify (insecure)
Vinaum8 commented 1 week ago

Shouldn't argocd update the golang lib?

maxkaemmerer commented 1 week ago

We switched over to password authentication until there is a fix. Not ideal but at least Argo keeps working.

If you do this keep in mind you also need to update the repoURL in your Application resources that access Azure Repos, not just inside the Repo Secret.

danijam commented 1 week ago

@rouke-broersma If we wanted to disable TLS verify (for the short term, acceptable for our current usage). Is there a way to-do that declaratively? I only see reference to the argo cli... We create our repo credential using the repo-creds secret type declaratively and not via the cli. I'm not finding anything that suggests I can do it that way?

jannfis commented 1 week ago

Hey folks, I think this change needs to be done on the user side.

According to Microsoft's blog post, section "Phase I: User Opt-in", people will have to generate new private/public key pairs and update both, Azure configuration and Argo CD repository config with this new key.

Have you tried generating a new key pair, with a supported algorithm?

evanmcneal commented 1 week ago

What helped us is to get into contact with the Microsoft support, mentioning this issue. This is a bad workaround and should be solved in ArgoCD/go-git, but as long as there's no workaround in place, the MS support is the only party that can help here.

Do you mind posting what you told them and the sev levels and stuff for them to help so quickly?

danijam commented 1 week ago

Hey folks, I think this change needs to be done on the user side.

According to Microsoft's blog post, section "Phase I: User Opt-in", people will have to generate new private/public key pairs and update both, Azure configuration and Argo CD repository config with this new key.

Have you tried generating a new key pair, with a supported algorithm?

@jannfis Yes we've taken all the required steps, reading this thread through I believe that is the consensus that even though people have correctly generated a new sha2 512 key and deployed to argo and deployed the public to Azure DevOps the brownouts are still impacting. I think the assumption is the handshake is defaulting to rsa-ssh because its on offer? (I might be wrong)

riiv-hexagon commented 1 week ago

Hey folks, I think this change needs to be done on the user side. According to Microsoft's blog post, section "Phase I: User Opt-in", people will have to generate new private/public key pairs and update both, Azure configuration and Argo CD repository config with this new key. Have you tried generating a new key pair, with a supported algorithm?

@jannfis Yes we've taken all the required steps, reading this thread through I believe that is the consensus that even though people have correctly generated a new sha2 512 key and deployed to argo and deployed the public to Azure DevOps the brownouts are still impacting. I think the assumption is the handshake is defaulting to rsa-ssh because its on offer? (I might be wrong)

yes even after new sha2 512 generated keys the issue remains