GITSYNC_SSH_KNOWN_HOSTS=false doesn't work?

[abh]

I set GITSYNC_SSH_KNOWN_HOSTS=false in my environment, but git-sync is still failing with a host verification error:

 HEAD HEAD^{}): exit status 128: { stdout: \"\", stderr: \"Host key verification failed.\\r\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.\" }","failCount":1}

  - env:
    - name: GITSYNC_REPO
      value: git@gitea-ssh.gitosis.svc.cluster.local:user/repo.git
    - name: GITSYNC_SSH_KNOWN_HOSTS
      value: "false"
    - name: GITSYNC_LINK
      value: a
    - name: GITSYNC_ROOT
      value: /git/static
    - name: GITSYNC_MAX_FAILURES
      value: "-1"
    - name: GITSYNC_PERIOD
      value: 2m
    - name: GITSYNC_HTTP_BIND
      value: :8071
    image: registry.k8s.io/git-sync/git-sync:v4.0.0

Separately but maybe related: I do have a /etc/git-secret/known_hosts file and I tested it with a regular ssh client, but something else seems to be wrong that git-sync isn't reading that file (or using the keys from it correctly).

[thockin]

Can you run with -v 9 and paste the "starting up" line (redacted as you see fit)? Specifically looking for "--ssh-known-hosts=" - when I run it, I see "false".

GITSYNC_SSH_KNOWN_HOSTS=false bin/linux_amd64/git-sync --root=/tmp/gs --repo=git@github.com:kubernetes/git-sync --depth 1 --ssh --ssh-key-file ~/.ssh/id_ed25519 -v 9 2>&1 | grep -i SSH
{"logger":"","ts":"2023-10-11 08:48:32.792496","caller":{"file":"main.go","line":722},"level":0,"msg":"starting up","pid":1754219,"uid":8119,"gid":89939,"home":"/home/thockin","flags":["--add-user=false","--change-permissions=0","--cookie-file=false","--depth=1","--exechook-backoff=3s","--exechook-timeout=30s","--git=git","--git-gc=always","--group-write=false","--help=false","--http-metrics=false","--http-pprof=false","--link=git-sync","--man=false","--max-failures=0","--max-sync-failures=0","--one-time=false","--password=REDACTED","--period=10s","--ref=HEAD","--repo=parse \"git@github.com:kubernetes/git-sync\": first path segment in URL cannot contain colon","--root=/tmp/gs","--ssh=true","--ssh-key-file=/home/thockin/.ssh/id_ed25519","--ssh-known-hosts=false","--ssh-known-hosts-file=/etc/git-secret/known_hosts","--stale-worktree-timeout=0s","--submodules=recursive","--sync-timeout=2m0s","--timeout=0","--v=-1","--verbose=9","--version=false","--wait=0","--webhook-backoff=3s","--webhook-method=POST","--webhook-success-status=200","--webhook-timeout=1s"]}
{"logger":"","ts":"2023-10-11 08:48:32.806215","caller":{"file":"main.go","line":1931},"level":1,"msg":"setting up git SSH credentials"}
{"logger":"","ts":"2023-10-11 08:48:32.806241","caller":{"file":"main.go","line":1953},"level":9,"msg":"setting GIT_SSH_COMMAND","value":"ssh -i /home/thockin/.ssh/id_ed25519 -o StrictHostKeyChecking=no"}

[abh]

oh, yeah -- that's weird (formatted with jq for readability)

{
  "logger": "",
  "ts": "2023-10-12 06:18:57.684869",
  "caller": {
    "file": "main.go",
    "line": 722
  },
  "level": 0,
  "msg": "starting up",
  "pid": 11,
  "uid": 0,
  "gid": 0,
  "home": "/tmp",
  "flags": [
    "--add-user=false",
    "--branch=main",
    "--change-permissions=0",
    "--cookie-file=false",
    "--depth=1",
    "--dest=a",
    "--exechook-backoff=3s",
    "--exechook-timeout=30s",
    "--git=git",
    "--git-gc=always",
    "--group-write=false",
    "--help=false",
    "--http-bind=:8071",
    "--http-metrics=false",
    "--http-pprof=false",
    "--link=a",
    "--man=false",
    "--max-failures=-1",
    "--max-sync-failures=-1",
    "--one-time=false",
    "--password=REDACTED",
    "--period=2m0s",
    "--ref=main",
    "--repo=parse \"git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git\": first path segment in URL cannot contain colon",
    "--root=/git/static",
    "--ssh=false",
    "--ssh-key-file=/etc/git-secret/ssh",
    "--ssh-known-hosts=false",
    "--ssh-known-hosts-file=/etc/git-secret/known_hosts",
    "--stale-worktree-timeout=0s",
    "--submodules=recursive",
    "--sync-timeout=10m0s",
    "--timeout=600",
    "--v=-1",
    "--verbose=0",
    "--version=false",
    "--wait=120",
    "--webhook-backoff=3s",
    "--webhook-method=POST",
    "--webhook-success-status=200",
    "--webhook-timeout=1s"
  ]
}

It doesn't pickup the VERBOSE option though it's set:

% kubectl -n ask exec -ti static-6c56dbff98-ngr8b  -- env | grep GITSYNC
Defaulted container "git-sync-askweb" out of: git-sync-askweb, git-sync-devcdn, git-sync-ntppool-news, apache
GITSYNC_REPO=git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git
GITSYNC_SSH_KNOWN_HOSTS=false
GITSYNC_VERBOSE=9
GITSYNC_LINK=a
GITSYNC_ROOT=/git/static
GITSYNC_MAX_FAILURES=-1
GITSYNC_PERIOD=2m
GITSYNC_SYNC_TIMEOUT=5m
GITSYNC_HTTP_BIND=:8071

So running it manually from the container, HTTP_BIND to another port:

# /git-sync -v=9
{"logger":"","ts":"2023-10-12 06:29:01.288796","caller":{"file":"main.go","line":558},"level":0,"msg":"setting --ref from deprecated --branch"}
{"logger":"","ts":"2023-10-12 06:29:01.288881","caller":{"file":"main.go","line":590},"level":0,"msg":"setting --link from deprecated --dest"}
{"logger":"","ts":"2023-10-12 06:29:01.288893","caller":{"file":"main.go","line":600},"level":0,"msg":"setting --period from deprecated --wait"}
{"logger":"","ts":"2023-10-12 06:29:01.288911","caller":{"file":"main.go","line":631},"level":0,"msg":"setting --sync-timeout from deprecated --timeout"}
{"logger":"","ts":"2023-10-12 06:29:01.288922","caller":{"file":"main.go","line":640},"level":0,"msg":"setting --max-failures from deprecated --max-sync-failures"}
{"logger":"","ts":"2023-10-12 06:29:01.288959","caller":{"file":"main.go","line":722},"level":0,"msg":"starting up","pid":88,"uid":0,"gid":0,"home":"/tmp","flags":["--add-user=false","--branch=main","--change-permissions=0","--cookie-file=false","--depth=1","--dest=a","--exechook-backoff=3s","--exechook-timeout=30s","--git=git","--git-gc=always","--group-write=false","--help=false","--http-bind=:8079","--http-metrics=false","--http-pprof=false","--link=a","--man=false","--max-failures=-1","--max-sync-failures=-1","--one-time=false","--password=REDACTED","--period=2m0s","--ref=main","--repo=parse \"git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git\": first path segment in URL cannot contain colon","--root=/git/static","--ssh=false","--ssh-key-file=/etc/git-secret/ssh","--ssh-known-hosts=false","--ssh-known-hosts-file=/etc/git-secret/known_hosts","--stale-worktree-timeout=0s","--submodules=recursive","--sync-timeout=10m0s","--timeout=600","--v=-1","--verbose=9","--version=false","--wait=120","--webhook-backoff=3s","--webhook-method=POST","--webhook-success-status=200","--webhook-timeout=1s"]}
{"logger":"","ts":"2023-10-12 06:29:01.289082","caller":{"file":"main.go","line":781},"level":2,"msg":"created private gitconfig file","path":"/tmp/git-sync.gitconfig.51095310"}
{"logger":"","ts":"2023-10-12 06:29:01.289115","caller":{"file":"main.go","line":2062},"level":5,"msg":"running command","cwd":"","cmd":"git config --global gc.autoDetach false"}
{"logger":"","ts":"2023-10-12 06:29:01.290460","caller":{"file":"main.go","line":2062},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.322153ms"}
{"logger":"","ts":"2023-10-12 06:29:01.290483","caller":{"file":"main.go","line":2062},"level":5,"msg":"running command","cwd":"","cmd":"git config --global gc.pruneExpire now"}
{"logger":"","ts":"2023-10-12 06:29:01.291856","caller":{"file":"main.go","line":2062},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.338313ms"}
{"logger":"","ts":"2023-10-12 06:29:01.291939","caller":{"file":"main.go","line":2062},"level":5,"msg":"running command","cwd":"","cmd":"git config --global credential.helper \"cache --timeout 3600\""}
{"logger":"","ts":"2023-10-12 06:29:01.292945","caller":{"file":"main.go","line":2062},"level":6,"msg":"command result","stdout":"","stderr":"","time":"950.256µs"}
{"logger":"","ts":"2023-10-12 06:29:01.292964","caller":{"file":"main.go","line":2062},"level":5,"msg":"running command","cwd":"","cmd":"git config --global safe.directory *"}
{"logger":"","ts":"2023-10-12 06:29:01.294313","caller":{"file":"main.go","line":2062},"level":6,"msg":"command result","stdout":"","stderr":"","time":"1.315651ms"}
{"logger":"","ts":"2023-10-12 06:29:01.294503","caller":{"file":"main.go","line":880},"level":0,"msg":"serving HTTP","endpoint":":8079","reasons":["liveness"]}
{"logger":"","ts":"2023-10-12 06:29:01.294538","caller":{"file":"main.go","line":1736},"level":3,"msg":"syncing","repo":"git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git"}
{"logger":"","ts":"2023-10-12 06:29:01.294557","caller":{"file":"main.go","line":1240},"level":3,"msg":"repo directory exists","path":"/git/static"}
{"logger":"","ts":"2023-10-12 06:29:01.294574","caller":{"file":"main.go","line":1312},"level":3,"msg":"sanity-checking git repo","repo":"/git/static"}
{"logger":"","ts":"2023-10-12 06:29:01.294649","caller":{"file":"main.go","line":1323},"level":5,"msg":"running command","cwd":"/git/static","cmd":"git rev-parse --show-toplevel"}
{"logger":"","ts":"2023-10-12 06:29:01.295555","caller":{"file":"main.go","line":1323},"level":6,"msg":"command result","stdout":"/git/static","stderr":"","time":"872.458µs"}
{"logger":"","ts":"2023-10-12 06:29:01.295573","caller":{"file":"main.go","line":1336},"level":5,"msg":"running command","cwd":"/git/static","cmd":"git fsck --no-progress --connectivity-only"}
{"logger":"","ts":"2023-10-12 06:29:01.299700","caller":{"file":"main.go","line":1336},"level":6,"msg":"command result","stdout":"","stderr":"notice: HEAD points to an unborn branch (git-sync)\nnotice: No default references","time":"4.101324ms"}
{"logger":"","ts":"2023-10-12 06:29:01.299725","caller":{"file":"main.go","line":1242},"level":4,"msg":"repo directory is valid","path":"/git/static"}
{"logger":"","ts":"2023-10-12 06:29:01.299737","caller":{"file":"main.go","line":1269},"level":5,"msg":"running command","cwd":"/git/static","cmd":"git remote get-url origin"}
{"logger":"","ts":"2023-10-12 06:29:01.300604","caller":{"file":"main.go","line":1269},"level":6,"msg":"command result","stdout":"git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git","stderr":"","time":"844.565µs"}
{"logger":"","ts":"2023-10-12 06:29:01.300627","caller":{"file":"main.go","line":1654},"level":5,"msg":"running command","cwd":"/git/static","cmd":"git ls-remote -q git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git main main^{}"}
The authenticity of host 'gitea-ssh.gitosis.svc.cluster.local (10.43.38.57)' can't be established.
RSA key fingerprint is SHA256:GkWcque9b57OZ+ZayHkytHjpjwW1GHi16waq70RgnWA.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
{"logger":"","ts":"2023-10-12 06:29:11.016297","caller":{"file":"main.go","line":974},"msg":"error syncing repo, will retry","error":"Run(git ls-remote -q git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git main main^{}): exit status 128: { stdout: \"\", stderr: \"Could not create directory '/root/.ssh' (No such file or directory).\\r\\nFailed to add the host to the list of known hosts (/root/.ssh/known_hosts).\\r\\ngit@gitea-ssh.gitosis.svc.cluster.local: Permission denied (publickey).\\r\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.\" }","failCount":1}
{"logger":"","ts":"2023-10-12 06:29:11.016363","caller":{"file":"main.go","line":1042},"level":3,"msg":"next sync","waitTime":"2m0s"}

(I typed yes by hand)

However, if I copy /etc/git-secret/ssh to /root/.ssh/id_rsa and /etc/git-secret/known_hosts to the same directory, the regular ssh client logs in accepting the host key as-is.

The command line git client, too:

# git clone git@gitea-ssh.gitosis.svc.cluster.local:ask/askweb.git
Cloning into 'askweb'...
remote: Enumerating objects: 8279, done.
remote: Counting objects: 100% (211/211), done.
remote: Compressing objects: 100% (175/175), done.
remote: Total 8279 (delta 114), reused 0 (delta 0), pack-reused 8068
Receiving objects: 100% (8279/8279), 146.67 MiB | 38.35 MiB/s, done.
Resolving deltas: 100% (4546/4546), done.
Updating files: 100% (2941/2941), done.

The hostkey is from gitea -- I wonder if

gitea-ssh.gitosis.svc.cluster.local ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8CjQyJb7MsUFwCIdFiUiymrThnkdc5+io168okjwKkp0ZWuKU5GYDVGUJsofARZAdOdslpfFVUB2tOPIHOjcboLNmPU9gAglkkoPrz+U7nVQYWzArfsnt3XjZIgEBZE9nWngmy1fQKXP20zwIQq6xeVDEKD3MjgyQoaIbSesy0g3h9TQya6okXA1T+cV+Hkpn1KwQSvbIOPX1vgybvG0RzyXb4bT5vpWUUE7epTSGwcw04a/MDF0qkW3C3EPjbUD5v2pBhVRSmbLxUAtzFn95nCAFXCZr1g0p1U+TVFe/U/L5UHKX+gLimda6bv4JcVDDAWQl73mklIHArtOZmCNzTI9rpSeqXaHk3PFzW2HOKAr5U9e+S8Ns6CfEzljz3hXnn5IVQSOkYCP80tuRxeuVIkUuKeUqtkL1ftxcl8vCX+2maSQRsy7S4gMFUqRO5e6I+dvD3DuUguumuL+5knSqyD6liXNjy5fmmWJSM+jX62RFhHoOYDGr0aORS7voKJ+DDnjGpB+/czzuwj1LdHFccYFIkbyiXaI2sq0DKXgJXcGIwpph8i235uv5UqBeBUWXIE6LaBJtic9/16oX1nsc8nh3Ug5/imF1p62ngf4L2x9xpUtn8l+9WNFn+9N7Fm1OCI9KtuMOCS3xlJkZzzyYTshEgik4pCG3phSD/rvE4Q==

[thockin]

I think I know what's up.

Note that it says "--ssh=false" - in all vertsionsup to and inclusing 4.0.0 you need to set --ssh or the GIT_SSH_COMMAND will not be set properly.

I made a change in master, which will go into the next release, which makes --ssh redundant. It will still work, but is not needed any more.

So please try setting that, and let me know, so we can close this :)

[abh]

oooh, nice — thank you, Tim.

This explains too why I thought I had 4.0.0 working briefly; and then it broke when I went through to v4 documentation and cleaned up the configuration (not realizing that 4.0.0 didn’t have the GITSYNC_SSH parameter optional).

Thanks again!

[thockin]

ACK. Will fix docs.