Adding SSH key using SSH_PRIVATE_KEY fails

Jasper-Ben commented 4 years ago

Environment

Docker image: kasproject/kas:2.2 SSH private key in PEM RSA format

Expected behavior

running SSH_PRIVATE_KEY=/path/to/ssh/key kas build /mnt/kas-foo.yml in the docker container should add the SSH key to the ssh-agent using ssh-add.

Actual behavior

builder@e76a2ef8b3c4:/tmp$ SSH_PRIVATE_KEY=/path/to/ssh/key kas build /mnt/kas-foo.yml 
2020-09-09 11:08:45 - INFO     - kas 2.2 started
2020-09-09 11:08:45 - INFO     - /mnt$ git rev-parse --show-toplevel
2020-09-09 11:08:45 - INFO     - adding SSH key
2020-09-09 11:08:45 - ERROR    - failed to add ssh key: b'Error loading key "(stdin)": invalid format\n'

Additional information

Manually running ssh-agent -s and ssh-add /path/to/ssh/key works:

builder@e76a2ef8b3c4:/tmp$ ssh-agent -s
SSH_AUTH_SOCK=/tmp/ssh-Ds4LFCbiHB34/agent.122; export SSH_AUTH_SOCK;
SSH_AGENT_PID=123; export SSH_AGENT_PID;
echo Agent pid 123;
builder@e76a2ef8b3c4:/tmp$ ssh-add /path/to/ssh/key
Identity added: /path/to/ssh/key (/path/to/ssh/key)

henning-schild commented 4 years ago

Note that you should not use issues but get in touch via the mailinglist instead. Thanks for reporting the problem, we will see if we can reproduce and fix it. But in fact you should probably raise it again on the list.

As a workaround you could use the --ssh-dir arg of kas-docker for the time being.

Jasper-Ben commented 4 years ago

Note that you should not use issues but get in touch via the mailinglist instead. Thanks for reporting the problem, we will see if we can reproduce and fix it. But in fact you should probably raise it again on the list.

As a workaround you could use the --ssh-dir arg of kas-docker for the time being.

Thanks for the info. I submitted a patch on the mailing list.

vivien commented 3 years ago

Is this supposed to be fixed in 2.5? Using the kas-container wrapper, I still get an Host key verification failed error while using any of SSH_PRIVATE_KEY=path/to/key/file, --ssh-dir path/to/key/, KAS_SSH_DIR=path/to/key/, or SSH_AUTH_SOCK=/tmp/ssh-XXXXXXABCde1/agent.123456.

For context, with KAS_SSH_DIR mounting the directory containing the key in /etc/skel/.ssh, I can successfully git clone git@github.com:company/project.git from the kas-container shell, but not with SSH_PRIVATE_KEY. With the build command, BitBake fails to fetch the sources from SRC_URI = "git://git@github.com/company/project.git;protocol=ssh" (translated to ssh://git@github.com/company/project.git).

jan-kiszka commented 3 years ago

It's --ssh-dir path/to/ssh-config-dir with kas-container.

vivien commented 3 years ago

I know Jan, please read again my second paragraph. I'm saying that it works with KAS_SSH_DIR from kas-container shell but not from kas-container build.

jan-kiszka commented 3 years ago

I'm still trying to sort what you are writing...

KAS_SSH_DIR should be equivalent to --ssh-dir irrespective of build vs. shell. You can also validate that (or prove me wrong) by using -d (verbose mode) and comparing the docker command lines.

SSH_PRIVATE_KEY is not supported with kas-container. You can quickly see that from scanning the script - no special handling. But that would be needed. And actually more because you would then still lack known_hosts settings.

vivien commented 3 years ago

Let me give you a bit more context:

I need to package an application from a private github repository https://github.com/company/project
For read access, I've generated a pair of keys, and I've added a read-only deploy key in the repository settings
With KAS_SSH_DIR=deploy-key-dir ./kas-container shell kas.yml I can successfully run git clone git@github.com:company/project.git
My foo.bb recipe contains SRC_URI = "git://git@github.com/company/project.git;protocol=ssh" (translated to ssh://git@github.com/company/project.git)
KAS_SSH_DIR=deploy-key-dir KAS_TARGET=foo ./kas-container build kas.yml fails with Host key verification failed

What are my options? Should I invoke kas manually, or maybe write a custom Dockerfile to add the key pair in ~/.ssh?

jan-kiszka commented 3 years ago

Could you compare/share first of all the resulting docker command lines as I suggested?

From my own setups I can only report that this feature works here as it is supposed to be.

Jasper-Ben commented 3 years ago

Let me give you a bit more context:

* I need to package an application from a private github repository _https://github.com/company/project_

* For read access, I've generated a pair of keys, and I've added a read-only [deploy key](https://docs.github.com/en/developers/overview/managing-deploy-keys#deploy-keys) in the repository settings

* With `KAS_SSH_DIR=deploy-key-dir ./kas-container shell kas.yml` I can successfully run `git clone git@github.com:company/project.git`

* My _foo.bb_ recipe contains `SRC_URI = "git://git@github.com/company/project.git;protocol=ssh"` (translated to _ssh://git@github.com/company/project.git_)

* `KAS_SSH_DIR=deploy-key-dir KAS_TARGET=foo ./kas-container build kas.yml` fails with _Host key verification failed_

What are my options? Should I invoke kas manually, or maybe write a custom Dockerfile to add the key pair in ~/.ssh?

AFAIK there has not been any change here since I openend the issue. I am also not sure what the right way to do this is. I've got it like this for now, but I'm sure that's not how it was intended to be done: https://github.com/iris-GmbH/iris-kas/blob/develop/docker-compose.yml

jan-kiszka commented 3 years ago

KAS_SSH_DIR=/ssh-dir kas-container -d shell/build kas.yml:

docker run -v /project-path:/repo:rw -v /project-path:/work:rw --workdir=/work -v /project-path/build:/build:rw -e KAS_BUILD_DIR=/build -e USER_ID=1000 -e GROUP_ID=100 --rm -v /ssh-dir:/etc/skel/.ssh:ro -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --privileged ghcr.io/siemens/kas/kas-isar:2.5 -d shell /repo/kas.yml

docker run -v /project-path:/repo:ro -v /project-path:/work:rw --workdir=/work -v /project-path/build:/build:rw -e KAS_BUILD_DIR=/build -e USER_ID=1000 -e GROUP_ID=100 --rm -v /ssh-dir:/etc/skel/.ssh:ro -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --privileged ghcr.io/siemens/kas/kas-isar:2.5 -d build /repo/kas.yml

The only difference is :ro/rw for repo. If that makes a difference can easily be checked as well (--repo-rw).

You likely want to debug into bitbake fetching (log files...?), why it complains about unknown host keys. Again, it works here, and I do not yet see what is different in your setup.

vivien commented 3 years ago

So do_fetch fails to non-interactively check the host key while running this ugly command:

export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="..."; export HOME="/tmp/tmpbajrfgro"; git -c core.fsyncobjectfiles=0 ls-remote ssh://git@github.com/company/private.git

But running it manually from the kas shell, I get prompted Are you sure you want to continue connecting (yes/no)?, and typing yes works fine.

For some reason, the private key from the original user home (not the exported tmp dir above) is considered, but not the user ssh config, so I had to configure the host key checking system-wide.

I ended up with the following Dockerfile which successfully fetches the private repository:

FROM ghcr.io/siemens/kas/kas:2.5
RUN sed -i -e "s,#   StrictHostKeyChecking ask,    StrictHostKeyChecking accept-new," /etc/ssh/ssh_config
RUN useradd -ms /bin/bash kas
COPY deploy_key /home/kas/.ssh/id_ed25519
RUN chown -R kas:kas /home/kas
USER kas
WORKDIR /home/kas/work
ENTRYPOINT ["kas"]
CMD ["build", "kas.yml"]

Invoked with:

docker run --rm -it -u $(id -u):$(id -g) -v "$(pwd):/home/kas/work:rw" company/kas

Honestly, I'm not sure what the source of the problem is, why a non-existent temporary home is exported, should the BitBake fetcher use StrictHostKeyChecking accept-new somehow, should ~/.ssh/config be considered, should kas do something about it, etc.

jan-kiszka commented 3 years ago

I didn't try if we have issues with ssh accepting a config passed in via ssh-dir, but passing in a valid known_hosts is obviously highly recommended in order to have a more secure CI setup. And this is how my setup looks like and why it works.

Jasper-Ben commented 3 years ago

COPY deploy_key /home/kas/.ssh/id_ed25519

Just be aware that you now have a SSH access key baked into your image, thus every user who gets hold of the image can access the private git repos (and possibly more) which should be avoided if possible.

Jasper-Ben commented 3 years ago

I've posted a patch on the mailing list, which addresses the "private key as file" usecase by introducing the SSH_PRIVATE_KEY_FILE variable: https://groups.google.com/g/kas-devel/c/BwGYUVMA-dw

vivien commented 3 years ago

I finally got rid of the kas-container script in favor of this ./kas wrapper below. It can successfully fetch Github private repositories listed with the form _SRCURI = "git://git@github.com/company/private.git;protocol=ssh".

#!/bin/sh
#
# https://kas.readthedocs.io/en/2.5/command-line.html

export KAS_WORK_DIR=${KAS_WORK_DIR:-$PWD}
export SSTATE_DIR=${SSTATE_DIR:-$KAS_WORK_DIR/sstate-cache}
export DL_DIR=${DL_DIR:-$KAS_WORK_DIR/downloads}

exec docker run --rm -it \
  -v $KAS_WORK_DIR:$KAS_WORK_DIR:rw \
  -e $KAS_WORK_DIR \
  -w $KAS_WORK_DIR \
  -v $SSTATE_DIR:$SSTATE_DIR:rw \
  -e SSTATE_DIR \
  -v $DL_DIR:$DL_DIR:rw \
  -e DL_DIR \
  -v ~/.ssh/known_hosts:/etc/skel/.ssh/known_hosts:ro \
  -v $SSH_AUTH_SOCK:$SSH_AUTH_SOCK:ro \
  -e SSH_AUTH_SOCK \
  -e KAS_MACHINE \
  -e KAS_DISTRO \
  -e KAS_TARGET \
  -e KAS_TASK \
  -e USER_ID=$(id -u) \
  -e GROUP_ID=$(id -g) \
  ghcr.io/siemens/kas/kas:2.5 "$@"

It obviously requires that the host machine uses ssh-agent, knows about the host to clone from (github.com in this case), and specifies a yaml file relative to the _KAS_WORKDIR.

etam commented 3 years ago

Based on snipped posted by vivien I found that this works as well:

kas-container --runtime-args "-v ${HOME}/.ssh/known_hosts:/etc/skel/.ssh/known_hosts:ro -v ${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK}:ro -e SSH_AUTH_SOCK" build ...

siemens / kas