search

ros-industrial / industrial_ci

Easy continuous integration repository for ROS repositories
Apache License 2.0
241 stars 128 forks source link

[gitlab] 'Error loading key "/dev/fd/63": invalid format' error #756

Open 130s opened 2 years ago

130s commented 2 years ago

Issue

On a private repo I keep getting this error (which has prevented me from switching to industrial_ci on Gitlab)

$ git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 47
Error loading key "/dev/fd/63": invalid format
could not add ssh key
Cleaning up file based variables 00:02
ERROR: Job failed: exit code 1

Attempted but doesn't seem to be working

Env

mathias-luedtke commented 2 years ago

I believe it's the same issue as https://gitlab.com/gitlab-examples/ssh-private-key/-/issues/1#note_48526556 And there they suggest echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add

130s commented 2 years ago

Thanks. I stopped seeing the exact error message above but I still see the following with plusone-robotics/industrial_ci/fix-gitlab-sshkey.

:
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 40
Error loading key "(stdin)": invalid format
could not add ssh key
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1
mathias-luedtke commented 2 years ago

So there might be as well something with actual key data. Without a public test cases, it hard to tell what's going wrong.

130s commented 2 years ago

Tested on a private repo on my personal org (which I don't mind giving an acecss), unlike the OP that is on a corporate org, I can't repro this issue. So I assume there's a difference b/w personal org and corporate org that ICI's Gitlab tool isn't yet handling.

Detail Same as OP, `SSH_PRIVATE_KEY` is supplied by using the repo's feature (Settings --> CI/CD --> Variables). ``` : $ git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master $ .industrial_ci/gitlab.sh ROS_DISTRO=melodic Agent pid 35 Identity added: /dev/fd/63 (/dev/fd/63) Running test 'source_tests' pull_docker_image 00:03 'pull_docker_image' returned with code '0' after 0 min 3 sec $ docker create --init --env-file /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/isolation/docker.env --rm -v /builds/130s/sample_private_repo:/builds/130s/sample_private_repo:ro -e TARGET_REPO_PATH=/builds/130s/sample_private_repo -v /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:ro -e ICI_SRC_PATH=/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src -v /tmp/ssh-XXXXXXoaMFJi/agent.34:/tmp/ssh-XXXXXXoaMFJi/agent.34:rw -e SSH_AUTH_SOCK=/tmp/ssh-XXXXXXoaMFJi/agent.34 -t --entrypoint -w /builds/130s/sample_private_repo ubuntu:bionic /bin/bash /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/run.sh source_tests run_source_tests Copy credentials: /root/.ssh $ docker start -a b49c0697fe94eb39c3a60f16f1a699791505c1f31d8aceff328d736c151f374b : Job succeeded ``` `.gitlab-ci.yml` ``` image: docker:git services: - docker:dind before_script: - apk add --update bash coreutils tar - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master build: script: .industrial_ci/gitlab.sh ROS_DISTRO=melodic ```

Interestingly, btw, I tested deleting SSH_PRIVATE_KEY on Gitlab from the same repo, which I'd expect ICI to fail with the same/similar error in this ticket as ssh key is now unavailable, but the job still went on and passed.

$ git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Running test 'source_tests'
pull_docker_image 00:03
'pull_docker_image' returned with code '0' after 0 min 3 sec
$ docker create --init --env-file /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/isolation/docker.env --rm -v /builds/130s/sample_private_repo:/builds/130s/sample_private_repo:ro -e TARGET_REPO_PATH=/builds/130s/sample_private_repo -v /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src:ro -e ICI_SRC_PATH=/builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src -t --entrypoint  -w /builds/130s/sample_private_repo ubuntu:bionic /bin/bash /builds/130s/sample_private_repo/.industrial_ci/industrial_ci/src/run.sh source_tests run_source_tests
$ docker start -a 1d1a767849b582e12d0df5cb1534796933b7999d49e10c90b973fd2c89a4d4ef
init
'init' returned with code '0' after 0 min 34 sec
catkin_tools_setup
:
130s commented 2 years ago

Content of SSH_PRIVATE_KEY is questionable. But in our org SSH_PRIVATE_KEY is maintained at the top org level and I don't have an access to it so I'm wondering how to check its format etc.

With ICI removing tr -d https://github.com/plusone-robotics/industrial_ci/commit/5c29a28478cf3dadba61798aef3e5989a5014662, the err msg changed from https://github.com/ros-industrial/industrial_ci/issues/756#issue-1046214448 (no longer seeing Error loading key "/dev/fd/63": invalid format).

https://gitlab.com/foo/baa/-/jobs/1798432233#L127

$ .industrial_ci/gitlab.sh ROS_DISTRO=melodic
Agent pid 35
Error loading key "(stdin)": invalid format
could not add ssh key
130s commented 2 years ago

So I assume there's a difference b/w personal org and corporate org that ICI's Gitlab tool isn't yet handling.

Spoke too soon. I can't say that yet. I have to suspect the key's validity as well as Mathias pointed out.

130s commented 1 year ago

This issue is no longer happening on my repos on gitlab.com although I don't think I've done anything specific (I have no idea if admins of our org has done anything to the admin env vars). Closing.

130s commented 1 year ago

Re-opening to continue discussion.

I still see the same issue using master of ICI. In order to get this issue passed I'm using our branch https://github.com/ros-industrial/industrial_ci/pull/806. So I'm still missing something.