ssh handshake failure when running "flux bootstrap git"

smveloso commented 2 years ago

Describe the bug

Upon running "flux bootstrap git" with a "--url" parameter pointing to an existing github repository, I get the error message:

✗ failed to clone repository: ssh: handshake failed: knownhosts: key mismatch

The github repository exists, is private and is accessible via "git clone ...".

The host key exists in "~/.ssh/known_hosts":

github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

There are no flux components running in the kubernetes cluster, this is the actual bootstraping. I have opted to use "bootstrap git" instead of "bootstrap github" mostly out of curiority.

If the host key is not present in "~/.ssh/known_hosts" I get a different (and quite expected) error message:

✗ failed to clone repository: ssh: handshake failed: knownhosts: key is unknown

It seems clear to me that whatever the problem is it's happening in the cli itself, nothing to do with the k8s cluster.

Steps to reproduce

Create a repository on github (e.g. acme)
Create or get access to a k8s cluster (kubeconfig at ~/.kube/acme)
Make sure github.com hostkey exists in "~/.ssh/known_hosts"
Run: flux bootstrap git --kubeconfig ~/.kube/acme --url=ssh://git@github.com/yourname/acme

The hostkey:

github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

Expected behavior

Correct bootstraping.

Screenshots and recordings

No response

OS / Distro

ArchLinux

Flux version

0.28.5

Flux check

flux --kubeconfig ~/.kube/flux-lab-one check ~/Env/fluxv2 ► checking prerequisites ✔ Kubernetes 1.22.7+k3s1 >=1.20.6-0 ► checking controllers ✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

smveloso commented 2 years ago

Refs I will probably need:

smveloso commented 2 years ago

Code:

https://github.com/smveloso/flux2/blob/smveloso/cmd/flux/bootstrap_git.go

then to:

https://github.com/smveloso/flux2/blob/smveloso/internal/bootstrap/git/gogit/gogit.go

smveloso commented 2 years ago

Using a self-hosted gitea instead of github, I can reproduce the error when the host key is of type ed25519.

git-infra.trt3.jus.br ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID/9BRYvWhc/zupJgEz09GtmneUqSAoFUojPOblDFB1F

smveloso commented 2 years ago

Problem appears to be (according to reports) on the ssh client in golang.

smveloso commented 2 years ago

Using this entry in "known_hosts" cloning worked:

github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=

smveloso commented 2 years ago

Indeed, the problem seems to happen only when the key type is ed25519.

But it is supported by github:

Adding [ECDSA](https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm) and [Ed25519](https://en.wikipedia.org/wiki/EdDSA#Ed25519) host keys for SSH

smveloso commented 2 years ago

According to https://github.com/go-git/go-git/issues/411#issue-1056130833 the problem may be related to precedence. The most preferred key type not being listed in the known_hosts file, the ssh client maybe refuses the lower precedence one that is listed.

smveloso commented 2 years ago

Problem is indeed located inside a go library:

File: .../go/pkg/mod/golang.org/x/crypto@v0.0.0-20210421170649-83a5a9bb288b/ssh/knownhosts/knownhosts.go

    // If the remote host starts using a different, unknown key type, we
    // also interpret that as a mismatch.
    if known, ok := knownKeys[remoteKey.Type()]; !ok || !keyEq(known.Key, remoteKey) {
        return keyErr
    }

This is very relevant: https://github.com/golang/go/issues/29286 and this is interesting: is this fixed by setting ClientConfig.HostKeyAlgorithms = []string{KeyAlgoED25519}?

smveloso-lab / flux2