search

canonical / cos-configuration-k8s-operator

This charmed operator for Kubernetes enables you to provide configurations to various components of the Canonical Observability Stack (COS) bundle.
https://charmhub.io/cos-configuration-k8s
Apache License 2.0
4 stars 5 forks source link

Syncing from private git repositories is broken #66

Closed przemeklal closed 12 months ago

przemeklal commented 1 year ago

Bug Description

I faced multiple issues trying to enable sync from a private repository.

  1. Specifying the private key during the charm deployment doesn't work, the private key file is not created. After resetting and setting again to the same value, the file is created.
  2. git-sync command fails because the known_hosts file path doesn't exist.

Now the following issues were encountered after fixing the above with juju ssh workarounds (all commands listed below).

  1. Remote server's SSH key is not auto-accepted/ignored by git-sync called by the charm and the sync action fails.
  2. Private SSH key has incorrect permissions 0644 instead of 0600 (or even more restrictive).

Additionally, there's no validation of the private key, after setting the option value using =$(cat id_ecdsa) results in a file without a newline at the end, using the =@id_ecdsa syntax works. Because of that, the sync action may also fail as it will report that the key file has an invalid format.

To Reproduce

  1. Deploy:

    juju deploy cos-configuration-k8s --config git_repo=git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted --config git_branch=main --config git_depth=1 --config git_ssh_key="$(cat redacted.key)" cos-configuration

  2. Try to sync:

    juju run-action cos-configuration/0 sync-now --wait
...
log:
- 2023-07-07 07:24:46 +0000 UTC Calling git-sync with --one-time...
- '2023-07-07 07:24:46 +0000 UTC ERROR: can''t configure SSH: can''t access SSH
key: stat /run/cos-config-ssh-key.priv: no such file or directory'
message: 'Sync error: Exited with code 1.'

  3. Reset the config option and set it to the same value again.

  4. Attempt to sync again, the key is there now but the action fails again:

    juju run-action cos-configuration/0 sync-now --wait
...
log:
- 2023-07-07 07:29:59 +0000 UTC Calling git-sync with --one-time...
- '2023-07-07 07:29:59 +0000 UTC ERROR: can''t configure SSH: can''t access SSH
known_hosts: stat /etc/git-secret/known_hosts: no such file or directory'
message: 'Sync error: Exited with code 1.'

  5. Created empty known_hosts file manually in an attempt to work this around:

    juju ssh --container git-sync cos-configuration/0 mkdir /etc/git-secret/
juju ssh --container git-sync cos-configuration/0 ls -l /etc/git-secret/known_hosts

  6. Synced again, this time it failed on Host key verification failed.

    juju run-action cos-configuration/0 sync-now --wait
...
log:
- 2023-07-07 07:40:05 +0000 UTC Calling git-sync with --one-time...
- 2023-07-07 07:40:05 +0000 UTC I0707 07:40:05.187060     146 main.go:473] "level"=0
"msg"="starting up" "pid"=146 "args"=["/git-sync","--repo","git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--ssh","--ssh-key-file","/run/cos-config-ssh-key.priv","--one-time"]
- 2023-07-07 07:40:05 +0000 UTC I0707 07:40:05.187213     146 main.go:923] "level"=0
"msg"="cloning repo" "origin"="git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted"
"path"="/git"
- '2023-07-07 07:40:05 +0000 UTC E0707 07:40:05.303971     146 main.go:525] "msg"="too
many failures, aborting" "error"="Run(git clone -v --no-checkout -b main --depth
1 git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted /git):
exit status 128: { stdout: "", stderr: "Cloning into ''/git''...\nHost key verification
failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you
have the correct access rights\nand the repository exists.\n" }" "failCount"=0'
message: 'Sync error: Exited with code 1.'

  7. I assumed git-sync prompts to accept the remote key so I ran the same command via juju ssh:

    juju ssh --container git-sync cos-configuration/0 "/git-sync --repo git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted --branch main --rev HEAD --depth 1 --root /git --dest repo --ssh --ssh-key-file /run/cos-config-ssh-key.priv --one-time"

    It did and I typed yes:

    The authenticity of host 'git.launchpad.net (185.125.188.44)' can't be established.
RSA key fingerprint is SHA256:UNOzlP66WpDuEo34Wgs8mewypV0UzqHLsIFoqwe8dYo.
Are you sure you want to continue connecting (yes/no)? yes

    It failed again after that:

    E0707 07:44:58.716917     181 main.go:525] "msg"="too many failures, aborting" "error"="Run(git clone -v --no-checkout -b main --depth 1 git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted /git): exit status 128: { stdout: "", stderr: "Cloning into '/git'...\nWarning: Permanently added 'git.launchpad.net,185.125.188.44' (RSA) to the list of known hosts.\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0644 for '/run/cos-config-ssh-key.priv' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\nLoad key \"/run/cos-config-ssh-key.priv\": bad permissions\r\nredacted@git.launchpad.net: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n" }" "failCount"=0

    The private key file created by the charm has incorrect permissions 0644 instead of 0600.

  8. I changed permissions manually using juju ssh to work this around:

    juju ssh --container git-sync cos-configuration/0 chmod 0600 /run/cos-config-ssh-key.priv

  9. Tried the sync again and it finally worked:

    juju run-action cos-configuration/0 sync-now --wait
unit-cos-configuration-0:
UnitId: cos-configuration/0
id: "46"
log:
- 2023-07-07 07:49:13 +0000 UTC Calling git-sync with --one-time...
- '2023-07-07 07:49:15 +0000 UTC Warning: I0707 07:49:13.172126     270 main.go:473]
"level"=0 "msg"="starting up" "pid"=270 "args"=["/git-sync","--repo","git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--ssh","--ssh-key-file","/run/cos-config-ssh-key.priv","--one-time"]'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:13.172225     270 main.go:923]
"level"=0 "msg"="cloning repo" "origin"="git+ssh://redacted@git.launchpad.net/~redacted/+git/redacted"
"path"="/git"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:13.886433     270 main.go:737]
"level"=0 "msg"="syncing git" "rev"="HEAD" "hash"="3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:14.839919     270 main.go:772]
"level"=0 "msg"="adding worktree" "path"="/git/3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"
"branch"="origin/main"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:14.845755     270 main.go:833]
"level"=0 "msg"="reset worktree to hash" "path"="/git/3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"
"hash"="3fecae3005ecf7eb23fd7e748b9999eb6ead91cb"'
- '2023-07-07 07:49:16 +0000 UTC Warning: I0707 07:49:14.845783     270 main.go:838]
"level"=0 "msg"="updating submodules"'
results:
git-sync-stdout: ""
status: completed
timing:
completed: 2023-07-07 07:49:19 +0000 UTC
enqueued: 2023-07-07 07:49:12 +0000 UTC
started: 2023-07-07 07:49:12 +0000 UTC

Environment

COS Lite on top of microk8s, charm from latest/edge.

Relevant log output

Included in steps to reproduce.

Additional context

No response

przemeklal commented 1 year ago

One more comment: the ssh private key must have a new line character at the end, otherwise git-sync will complain about the invalid format of the key and the action will fail. The typical workflow of juju config git_ssh_key=@id_rsa doesn't work, you need to cat the file, enter a newline manually and pass it to juju.