search

fluxcd / flux2

Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.
https://fluxcd.io
Apache License 2.0
6.46k stars 597 forks source link

Oracle VBS (Git host) doesn't work with Flux anymore due to SSH key type #4314

Closed ajhindle closed 6 months ago

ajhindle commented 1 year ago

Describe the bug

I can't run Flux reconcile against my repo in Oracle Visual Builder Studio (Git host) anymore. Before September 2023 it worked fine, but in the VBS latest update I think they made it so only SHA2 RSA keys work now. VBS don't support ECDSA keys, only RSA.

flux reconcile kustomization flux-system --with-source
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'ssh://idcs-abc.cicd/@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git': ssh: handshake failed: ssh: no common algorithm for host key; client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256]'

Please note the message: "client offered: [ssh-rsa], server offered: [rsa-sha2-512 rsa-sha2-256]"

Steps to reproduce

  1. Delete existing Flux key.
  2. Create new Flux key:
    flux create secret git flux-system --url=ssh://idcs-abc.cicd/@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git --ssh-key-algorithm rsa --ssh-rsa-bits=4096
  3. Reconcile: flux reconcile kustomization flux-system --with-source

Expected behavior

Expect the reconciliation to work without key handshake issues.

Screenshots and recordings

No response

OS / Distro

Ubuntu 20.04

Flux version

v2.1.1

Flux check

► checking prerequisites ✔ Kubernetes 1.27.4-eks-2d98532 >=1.25.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.35.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.1 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

Oracle Visual Builder Studio (VBS) in Oracle Cloud (OCI)

Container Registry provider

No response

Additional context

No response

Code of Conduct

ajhindle commented 1 year ago

Here is the known_host key from the flux-system secret:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
hiddeco commented 1 year ago

Can you share the output of ssh-keyscan for that repository URL?

ajhindle commented 1 year ago

Ok - like this?

ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
makkes commented 1 year ago

Your host only sends you an RSA host key but doesn't present that during the SSH handshake. Are you able to clone repos manually from that host?

ajhindle commented 1 year ago

Yes, I can clone manually with other SSH keys I've made.

ajhindle commented 1 year ago

Also, the SSH key that Flux created and added to the flux-system secret also works fine for a manual clone.

stefanprodan commented 1 year ago

Grep the known_hosts file on your machine and see what fingerprints are in there for that host. I guess you have SHA-2 keys in there.

makkes commented 1 year ago

Judging by all the info available now I'm pretty sure it's a server misconfiguration that you can mitigate by manually updating the flux-system Secret with the correct host key that you should be able to gather from your local known_hosts file as Stefan suggested.

ajhindle commented 1 year ago

The only differences between my local known_hosts and the known_hosts Flux made in the secret are:

The SSH key is the same in both.

The local's hashed host value won't work in the secret, right? I tried it anyway - no luck.

In Oracle VBS, when running Git commands with SSH, you seem to need to put the user name in also (see doco) - maybe that's part of the issue?
The flux secret doesn't know who the user is, AFAIK

ajhindle commented 1 year ago

Stefan asked me to run ssh-keyscan in a test pod (running linuxserver/openssh-server) on the EKS cluster running Flux. Did that, got the same result as when I run it locally.:

debug-shell:/# ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0

I also ran ssh -Q key from the test pod and got this output:

ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com
ssh-dss
ssh-dss-cert-v01@openssh.com
ssh-rsa
ssh-rsa-cert-v01@openssh.com
ajhindle commented 1 year ago

What command can I use to determine if the key is SHA1 or SHA2 ?

RE: what I wrote before "The flux secret doesn't know who the user is, AFAIK" I realised that the user name is stored in the "GitRepository" component, field "URL" i.e. ssh://username@repository

ajhindle commented 1 year ago

The Oracle team asks "which SSH library is Flux running? Is it OpenSSH?"

stefanprodan commented 1 year ago

We use https://github.com/go-git/go-git

ajhindle commented 10 months ago

Hi.

Today I have:

  1. Run git clone from Oracle VBS Git, from the same EKS cluster in a bitnami/git pod - the clone works fine with GIT_SSH_COMMAND="ssh -i ~/.ssh/prv.key" git clone ssh://idcs-abc.cicd@vbs-mel.developer.ocp.oraclecloud.com/vbs-mel_continuous-deployment_343/cd-bfs.git.
  2. Run flux create secret git with a regular RSA SHA2 key - this creates a secret in the EKS cluster with private key, public key and known_hosts data. The known_hosts section is not hashed. This produces the same error in the original issue description. I also tried replacing the known_hosts data with the hashed code from (1) - nothing changed, same end result.

Do we still think it's a server misconfiguration?

Is there a way to alter the SSH config file that Flux uses?

ajhindle commented 8 months ago

This issue appears to have been resolved by Oracle, I guess?
I can run bootstrap and reconcile with VBS Git fine now. When Flux couldn't SSH to VBS Git previously, the ssh-keyscan command returned fewer lines, fewer keys - see below results compared to previous output.

ssh-keyscan <redacted>.developer.ocp.oraclecloud.com
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAs/BWeDtLznfcGe0qI4jawP+bqhKYW7h2DoBLbLNf9UN4jjShsZZirJWuL/lLVd7s3PJl9hQKzchmdAa2ZERc89r1btPMq53n6+QnKAbXN5Bl7hi8iaei3mCgIjFbAga6tjwz4Hm64IwawcTzMSjue/QVKFX+Hod/5v1Ww6jnfIP7kAcqg6zY1h4F11XH4Yi294r+UuCSmTFqNciVs2h6QsHlquEl/VwNylwfEUCct8s/ABRJoDAwI05KvgLki4mx1dB2aVMSHHkusTQJwEsAU0+jDDHDu6dEoG98IUWirhjdMzDJ9iodImK7WskHdbfEWViCXnR0r7QqUpDXc0HONzyanpbCFhEYNYZtW/QtGDdyjo7iRyyilR0FfKfgZuHRVZipPEK9STDwTUxCC4bfKsS0UwqW2MO6pY/SjPDuhpDwFkODH7FjZrA9T7yQb8K/soOb78akFcA1qeZwOXPX+dePdcg50LwFBLh/H402iemU7nMHYwvsMkoiovGgRt1BgCdVehEIXve0aO1DKwGp6rDl8XVkzSfzRPgmVtjYYXHVjOeEVf7lEIVz+3ABpHZcsnAfcxR8BL2kGAyaHPr95JK1iqcxLczGPcXxTov61Os7cfHhqIWbl9ZlZgjYk8790XeFcTRD1W0j1OoWwE/Bt+NtkaWBOYTRkVlbanbMqSE=
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAG0aWG2uzvzrkfs0PX4LKquwUjX6zZJ0bSNWv8x2eQfl37/FufIDKn3CiHG4B62dJaifTqIZ0rzCj+kjqU4yJbNBwCjJUdlYykkRGN3Zx8Nhtlft7cDUWP0EBEMINUiRt5YVfio7A0vPXIy7mSsk3K45C/HFhZdUpI0WS6NqlIlnX65YA==
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
# <redacted>.developer.ocp.oraclecloud.com:22 SSH-2.0-APACHE-SSHD-2.8.0
<redacted>.developer.ocp.oraclecloud.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP8N61DYwPVAIohNDIeI/fT+6d/C+V81ErnEnmr+qFa6