Failed to checkout internal repos within GitLab CI #128

Closed joergweichelt-imm closed 5 days ago

I got the following Gitlab pipeline:

build-job:

image: ghcr.io/siemens/kas/kas:4.4

stage: build

variables:
KAS_PREMIRRORS: ssh://git@myserver.de https://gitlab-ci-token:${CI_JOB_TOKEN}@myserver.de/

script:
- kas build --update kas-project.yml

After checking out my internal repos, kas produces follwoing error:

2024-09-02 13:43:31 - ERROR    - bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/builds/imm/twall/yocto/twall-controlbox/layers/poky/scripts:/builds/imm/twall/yocto/twall-controlbox/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/twall-data/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/builds/imm/twall/yocto/twall-controlbox/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/twall-data/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot/usr/bin/crossscripts:/builds/imm/twall/yocto/twall-controlbox/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/twall-data/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/sbin:/builds/imm/twall/yocto/twall-controlbox/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/twall-data/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin:/builds/imm/twall/yocto/twall-controlbox/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/twall-data/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/sbin:/builds/imm/twall/yocto/twall-controlbox/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/twall-data/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/bin:/builds/imm/twall/yocto/twall-controlbox/layers/poky/bitbake/bin:/builds/imm/twall/yocto/twall-controlbox/build/tmp/hosttools"; export HOME="/tmp/tmp26snit_r"; git -c gc.autoDetach=false -c core.pager=cat ls-remote ssh://git@myserver.de:/internalrepo.git  failed with exit code 128, output:

2024-09-02 13:43:31 - ERROR    - git@myserver.de: Permission denied (publickey).
2024-09-02 13:43:31 - ERROR    - fatal: Could not read from remote repository.
2024-09-02 13:43:31 - ERROR    - 

2024-09-02 13:43:31 - ERROR    - Please make sure you have the correct access rights

2024-09-02 13:43:31 - ERROR    - and the repository exists.

2024-09-02 13:43:31 - ERROR    - 

2024-09-02 13:43:31 - ERROR    - The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

2024-09-02 13:43:31 - ERROR    -

It seems to me that after using the PREMIRRORS rewrite from ssh to https for checking out, KAS uses ssh for got ls-remote?

Hmm, this rather looks like bitbake itself is complaining here. And KAS_PREMIRRORS only works for kas itself. To adjust bitbake's view, regular PREMIRRORS comes into play.

Setting PREMIRRORS the same way like KAS_PREMIRRORS (in variables section) seems to have no effect. Is there anotherway to do this? OR Is there a better way beside using PREMIRRORS to work with repos in the CI script without using a private ssh key?

Hi! Since kas 4.4 there is no need to set anything related to git rewrites on gitlab CI. For details, see https://kas.readthedocs.io/en/latest/userguide/credentials.html#git-configuration

Please also check for a line Running on GitLab CI, to confirm kas correctly detected the gitlab environment. By using the kas shell -c <command> command, you can also inspect the .netrc file (which is used to inject the credentials / CI_JOB_TOKEN), as well as the .gitconfig.

'Running on GitLab CI' is present, yet without KAS_PREMIRRORS, kas tries to clone my repo using ssh and fails at this step already.

I tried kas shell -c 'cat .netrc' project.yml, but got a 'file not found'.

I think, kas's ssh->https rewrites doesnt come into play here because of .ssh/known_hosts exists.

If I dont create .ssh/known_hostsI finally see the log message Adding GitLab CI ssh -> https rewrites. Nevertheless, kas is going to clone my repos using ssh.

If I dont create .ssh/known_hosts finally see the log message Adding GitLab CI ssh -> https rewrites

Ok, this is also stated in the documentation, but maybe we should stress it a bit more: There must be no SSH configuration.

Nevertheless, kas is going to clone my repos using ssh.

Can you re-try the kas shell command to inspect the .gitconfig and .netrc? For me it looks like your git remote lines simply don't match the insteadof.

I'm unable to invoke any shell command as long as my repo is configured to be using a ssh URL in project.yml. (This was the reason to include known_hosts and KAS_PREMIRROR).

If I change the repo to use a https URL, I can cat ~/.gitconfig:

[user]
    email = kas@example.com
    name = kas User
[url "https://myserver.de/"]
    insteadOf = git@myserver.de:
    insteadOf = ssh://git@myserver.de/

(Note: cannot permantely change this since building outside Gitlab uses ssh + private key)

.netrc contains:

machine myserver.de
login gitlab-ci-token
password [MASKED]

After that, kas fails while trying to parse the layer from my recipe (FileNotFound: .../layer.conf) which seems a little strange because building the layer outside of Gitlab works fine. I'm going to dive into this...

The issue mentioned in my last comment was on my side... :-( So the problem remains that rewrite seems not to be get in affect: when cloning my repo, kas is trying to use ssh (and fails), when changing the repo config to use https, bitbake is using ssh later on (and fails).

After digging into this for a while it seemed I found the cause: my ssh URL contains a port, so I access my repo via ssh://git.myserver.de/2222. The rewrite rule kas created doesnt contain the port, so the rule doesnt match and the rewrite doesnt work,

Workaround for me is to manually create a .gitconfig containing the modified kas rule:

[url "https://myserver.de/"]
    insteadOf = git@myserver.de:
    insteadOf = ssh://git@myserver.de:2222/

and copy this file to ~/.gitconfig before running kas. And so far this seems to work.

P.S.: Putting git config --global url."https://gitlab.myserver.de".insteadof "ssh://git@gitlab.myserver.de:2222" in before_script may be a simpler approach.

Hi, thanks for the heads up.

my ssh URL contains a port

This detail was unfortunately not part of the issue description, but I also was not aware that this actually makes a difference. In general, I recommend to use the ssh config for the port mapping and always use a URI without the port. This is also required for tools like scp.

git config --global url."https://gitlab.myserver.de".insteadof "ssh://git@gitlab.myserver.de:2222"

Is there a way to automatically detect this using kas? The code that adds the insteadof is in [1], but while reading it I'm wondering if we better use CI_SERVER_SHELL_SSH_HOST and add an additional rule with CI_SERVER_SHELL_SSH_PORT appended.

[1] https://github.com/siemens/kas/blob/4e26e2189e4a7a7a9013ffa0cd9ffd8d8041361f/kas/libcmds.py#L300

@joergweichelt-imm I just implemented a fix to also handle the non-standard port cases correctly. It would be great if you could give it a try:

Commit: https://github.com/fmoessbauer/kas/commit/0e61f73051e02ebefa88c2094b99cfc6710722d2
Container: ghcr.io/fmoessbauer/kas/kas:next

Yes, please. Drop a note here or comment directly on https://groups.google.com/g/kas-devel/c/FdP4cr_9adM.

@fmoessbauer Your fix solved my problem, thank you very much! My CI script now runs without the need to manually add rewrite rules. And sorry for not mentioned the non-default port in the beginning ;-)

@jan-kiszka I can not connect to Google groups due to network restrictions :-(

siemens / kas

Failed to checkout internal repos within GitLab CI #128