search

siemens / kas

Setup tool for bitbake based projects
MIT License
339 stars 145 forks source link

Bitbake recipe problems when fetching private repositories in CI/CD with `KAS_PREMIRRORS` #121

Closed redeexpressos closed 1 month ago

redeexpressos commented 1 month ago

Trying to create a gitlab CI/CD pipeline.

  stage: build
  image: ghcr.io/siemens/kas/kas:latest
  timeout: 24 hours
  variables:
    KAS_ALLOW_ROOT: "yes"
    KAS_PREMIRRORS: "git@my.org.com: https://my.org.com/git/"
    DEBIAN_FRONTEND: "noninteractive"
  script:
    - git config --global credential.helper store
    - echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@my.org.com/git" > ~/.git-credentials
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@my.org.com/git".insteadOf ssh://git@my.org.com:1337
    - apt update && apt install chrpath cpio cpp g++ gawk gcc -y
    - kas build $PLATFORM

It looks like it fetches my private repos, however, when starting to build, something goes crazy:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)
$ kas build $PLATFORM
2024-06-03 15:59:27 - INFO     - kas 4.4 started
2024-06-03 15:59:27 - INFO     - Running on GitLab CI
2024-06-03 15:59:27 - INFO     - Cloning repository poky
2024-06-03 15:59:27 - INFO     - Cloning repository meta-openembedded
2024-06-03 15:59:27 - INFO     - Cloning repository meta-security
2024-06-03 15:59:27 - INFO     - Cloning repository meta-mono
2024-06-03 15:59:27 - INFO     - Cloning repository meta-rust-bin
2024-06-03 15:59:27 - INFO     - Cloning repository meta-freescale
2024-06-03 15:59:27 - INFO     - Cloning repository meta-freescale-3rdparty
2024-06-03 15:59:27 - INFO     - Cloning repository meta-toradex-bsp-common
2024-06-03 15:59:27 - INFO     - Cloning repository meta-toradex-nxp
2024-06-03 15:59:28 - INFO     - Repository meta-rust-bin already contains master as commit
2024-06-03 15:59:29 - INFO     - Repository meta-security updated
2024-06-03 15:59:29 - INFO     - Repository meta-freescale-3rdparty updated
2024-06-03 15:59:30 - INFO     - Repository meta-mono updated
2024-06-03 15:59:34 - INFO     - Repository meta-freescale updated
2024-06-03 15:59:38 - INFO     - Repository meta-toradex-nxp updated
2024-06-03 15:59:49 - INFO     - Repository meta-openembedded updated
2024-06-03 16:00:01 - INFO     - Repository poky updated
2024-06-03 16:00:33 - INFO     - Repository meta-toradex-bsp-common updated
2024-06-03 16:00:35 - INFO     - Repository poky checked out to e139e9d0ce343ba77a09601a976c92acd562c9df
2024-06-03 16:00:36 - INFO     - Repository meta-openembedded checked out to 0560b848996a0feb410a8cd8ca07c60fe2f3b5bc
2024-06-03 16:00:36 - INFO     - Repository meta-org checked out to 8e150c578066539161988aba077de6bd092ec540
2024-06-03 16:00:37 - INFO     - Repository meta-security checked out to 353078bc06c8b471736daab6ed193e30d533d1f1
2024-06-03 16:00:37 - INFO     - Repository meta-mono checked out to c4e9840412b32108fe009d62366e3a81f4d0aadd
2024-06-03 16:00:37 - INFO     - Repository meta-rust-bin checked out to 14e76cfdc4ce2d588ad60e05d5359f31d6fda709
2024-06-03 16:00:37 - INFO     - Repository meta-freescale checked out to fb17bfb8edcc9560bc1beb966a68f1f4c08ecfb3
2024-06-03 16:00:37 - INFO     - Repository meta-freescale-3rdparty checked out to 9e94b64bdfebcf7bfdf2af6447cec866a4efa814
2024-06-03 16:00:37 - INFO     - Repository meta-toradex-bsp-common checked out to 0cb7188573584dd9db39e5689e843edbe9440588
2024-06-03 16:00:37 - INFO     - Repository meta-toradex-nxp checked out to f45854580fd016013686d5f8a27770175dfb0025
2024-06-03 16:00:41 - INFO     - WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
2024-06-03 16:00:42 - INFO     - Loading cache...done.
2024-06-03 16:00:42 - INFO     - Loaded 0 entries from dependency cache.
2024-06-03 16:00:50 - ERROR    - ERROR: ExpansionError during parsing /builds/git/openembedded/builder/build/../layers/meta-org/recipes-org/abc-webserver/org-abc.bb
2024-06-03 16:00:50 - ERROR    - Traceback (most recent call last):
2024-06-03 16:00:50 - ERROR    - File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/__init__.py", line 1214, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, name='default'):
2024-06-03 16:00:50 - ERROR    - if srcrev == "AUTOINC":
2024-06-03 16:00:50 - ERROR    - >        srcrev = ud.method.latest_revision(ud, d, name)
2024-06-03 16:00:50 - ERROR    - 
2024-06-03 16:00:50 - ERROR    - File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/__init__.py", line 1626, in GitSM.latest_revision(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, name='default'):
2024-06-03 16:00:50 - ERROR    - except KeyError:
2024-06-03 16:00:50 - ERROR    - >            revs[key] = rev = self._latest_revision(ud, d, name)
2024-06-03 16:00:50 - ERROR    - return rev
2024-06-03 16:00:50 - ERROR    - File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/git.py", line 789, in GitSM._latest_revision(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, name='default'):
2024-06-03 16:00:50 - ERROR    - 
2024-06-03 16:00:50 - ERROR    - >        output = self._lsremote(ud, d, "")
2024-06-03 16:00:50 - ERROR    - # Tags of the form ^{} may not work, need to fallback to other form
2024-06-03 16:00:50 - ERROR    - File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/git.py", line 772, in GitSM._lsremote(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, search=''):
2024-06-03 16:00:50 - ERROR    - bb.fetch2.check_network_access(d, cmd, repourl)
2024-06-03 16:00:50 - ERROR    - >            output = runfetchcmd(cmd, d, True)
2024-06-03 16:00:50 - ERROR    - if not output:
2024-06-03 16:00:50 - ERROR    - File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/__init__.py", line 929, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/builds/git/openembedded/builder/layers/poky/scripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot/usr/bin/crossscripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/bin:/builds/git/openembedded/builder/layers/poky/bitbake/bin:/builds/git/openembedded/builder/build/tmp/hosttools"; export HOME="/tmp/tmpslgwc3rj"; git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat ls-remote ssh://git@my.orgcom:1337/abc.git ', d=<bb.data_smart.DataSmart object at 0x7fc5b5e65e90>, quiet=True, cleanup=[], log=None, workdir=None):
2024-06-03 16:00:50 - ERROR    - 
2024-06-03 16:00:50 - ERROR    - >        raise FetchError(error_message)
2024-06-03 16:00:50 - ERROR    - 
2024-06-03 16:00:50 - ERROR    - bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/builds/git/openembedded/builder/layers/poky/scripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot/usr/bin/crossscripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/bin:/builds/git/openembedded/builder/layers/poky/bitbake/bin:/builds/git/openembedded/builder/build/tmp/hosttools"; export HOME="/tmp/tmpslgwc3rj"; git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat ls-remote ssh://git@my.org.com:1337/abc.git  failed with exit code 128, output:
2024-06-03 16:00:50 - ERROR    - warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
2024-06-03 16:00:50 - ERROR    - remote: The project you were looking for could not be found or you don't have permission to view it.
2024-06-03 16:00:50 - ERROR    - fatal: repository 'https://my.org.com/git/abc.git/' not found
2024-06-03 16:00:50 - ERROR    - 
2024-06-03 16:00:50 - ERROR    - The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH
2024-06-03 16:00:50 - ERROR    - 
2024-06-03 16:00:50 - ERROR    - ERROR: Parsing halted due to errors, see error messages above
2024-06-03 16:00:51 - INFO     - Parsing recipes...
2024-06-03 16:00:51 - INFO     - Summary: There was 1 WARNING message.
2024-06-03 16:00:51 - INFO     - Summary: There were 2 ERROR messages, returning a non-zero exit code.
2024-06-03 16:00:51 - ERROR    - Command "/builds/git/openembedded/builder/build$ /builds/git/openembedded/builder/layers/poky/bitbake/bin/bitbake -c build org-image" failed
--- Error summary ---
ERROR: ExpansionError during parsing /builds/git/openembedded/builder/build/../layers/meta-org/recipes-org/abc-webserver/org-abc.bb
Traceback (most recent call last):
  File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/__init__.py", line 1214, in srcrev_internal_helper(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, name='default'):
         if srcrev == "AUTOINC":
    >        srcrev = ud.method.latest_revision(ud, d, name)

  File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/__init__.py", line 1626, in GitSM.latest_revision(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, name='default'):
             except KeyError:
    >            revs[key] = rev = self._latest_revision(ud, d, name)
                 return rev
  File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/git.py", line 789, in GitSM._latest_revision(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, name='default'):

    >        output = self._lsremote(ud, d, "")
             # Tags of the form ^{} may not work, need to fallback to other form
  File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/git.py", line 772, in GitSM._lsremote(ud=<bb.fetch2.FetchData object at 0x7fc5b804a1d0>, d=<bb.data_smart.DataSmart object at 0x7fc5b74514d0>, search=''):
                     bb.fetch2.check_network_access(d, cmd, repourl)
    >            output = runfetchcmd(cmd, d, True)
                 if not output:
  File "/builds/git/openembedded/builder/layers/poky/bitbake/lib/bb/fetch2/__init__.py", line 929, in runfetchcmd(cmd='export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/builds/git/openembedded/builder/layers/poky/scripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot/usr/bin/crossscripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/bin:/builds/git/openembedded/builder/layers/poky/bitbake/bin:/builds/git/openembedded/builder/build/tmp/hosttools"; export HOME="/tmp/tmpslgwc3rj"; git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat ls-remote ssh://git@my.org.com:1337/abc.git ', d=<bb.data_smart.DataSmart object at 0x7fc5b5e65e90>, quiet=True, cleanup=[], log=None, workdir=None):

    >        raise FetchError(error_message)

bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Fetch command export PSEUDO_DISABLED=1; export GIT_PROXY_COMMAND="oe-git-proxy"; export NO_PROXY="*"; export PATH="/builds/git/openembedded/builder/layers/poky/scripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot/usr/bin/crossscripts:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/usr/bin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/sbin:/builds/git/openembedded/builder/build/tmp/work/cortexa9t2hf-neon-poky-linux-gnueabi/org-abc/fetcheravoidrecurse-fetcheravoidrecurse/recipe-sysroot-native/bin:/builds/git/openembedded/builder/layers/poky/bitbake/bin:/builds/git/openembedded/builder/build/tmp/hosttools"; export HOME="/tmp/tmpslgwc3rj"; git -c core.fsyncobjectfiles=0 -c gc.autoDetach=false -c core.pager=cat ls-remote ssh://git@my.org.com:1337/abc.git  failed with exit code 128, output:
warning: core.fsyncObjectFiles is deprecated; use core.fsync instead
remote: The project you were looking for could not be found or you don't have permission to view it.
fatal: repository 'https://my.org.com:1337/abc.git' not found
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH
ERROR: Parsing halted due to errors, see error messages above
2024-06-03 16:00:51 - ERROR    - Command "/builds/git/openembedded/builder/layers/poky/bitbake/bin/bitbake -c build my-image" failed with error 1
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

However, locally, I can build it fine.

Is this a known issue? Am I missing something?

jan-kiszka commented 1 month ago

Did you already read https://kas.readthedocs.io/en/latest/userguide/credentials.html? It says, e.g., that you should not have to define your own .gitconfig in gitlab-ci.

Furthermore, why are you installing extra packages? The kas container should provide everything that OE needs.

redeexpressos commented 1 month ago

Eh looks like the problem is the same as here. I need to add permissions to the other repositories to be accessed by this one.

Furthermore, why are you installing extra packages? The kas container should provide everything that OE needs.

Forgot to update my pipeline regarding last issue :) But thanks.