Build example image in offline mode

[SeyerleinRichard]

Hallo,

I have to build a layer based on meta-iot2050, but have the requirement to be able to build it in an offline environment. As I failed to do so, I tried he same with meta-iot2050 to make sure my problem is not related with the definition of my own layer. Doing this I observed the same problems. So here's a summary of what I did:

First I added a file kas/opt/offline.yml to the project to set the necessary environment variables. The file's content is as follows:

header: 
  version: 10  

local_conf_header:  
  offline: |
    ISAR_USE_CACHED_BASE_REPO = "1"
    BB_NO_NETWORK = "1"

After doing this (and after doing a successful online build), I executed ./kas-container build kas-iot2050-example.yml:kas/opt/offline.yml. This build failed because the Isar recipe base_apt (in directory /meta/recipes-devtools/base-apt/base-apt.bb) expects the variables DISTRO and HOST_DISTRO to be identical (and of the form debian-[DistroName]). In meta-iot2050, DISTRO is set to iot2050-debian, so base_apt.bb failed for me.

I then modified the _baseapt.bb recipe in order to accept the iot2050-debian distro - at first manually, but checking for offline build and existence of the file, and then applying a patch in kas-container would certainly be possible, so there should be no problem here.

After doing this, the build still fails with the following output:

ERROR: sbuild-chroot-host-1.0-r0 do_rootfs_install: ExecutionError('/build/tmp/work/iot2050-debian-arm64/sbuild-chroot-host/1.0-r0/temp/run.rootfs_install_pkgs_download.23820', 100, None, None)
ERROR: Logfile of failure stored in: /build/tmp/work/iot2050-debian-arm64/sbuild-chroot-host/1.0-r0/temp/log.do_rootfs_install.23820
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_rootfs_install
| DEBUG: Executing shell function root_cleandirs
| DEBUG: Shell function root_cleandirs finished
| DEBUG: Executing shell function rootfs_prepare
| DEBUG: Shell function rootfs_prepare finished
| DEBUG: Executing shell function rootfs_configure_apt
| DEBUG: Shell function rootfs_configure_apt finished
| DEBUG: Executing shell function rootfs_do_mounts
| DEBUG: Shell function rootfs_do_mounts finished
| DEBUG: Executing shell function rootfs_install_pkgs_update
| Reading package lists...
| W: Unable to read /etc/apt/-/ - DirectoryExists (2: No such file or directory)
| W: Unable to read /etc/apt/sources.list.d/isar-apt.list - RealFileExists (2: No such file or directory)
| DEBUG: Shell function rootfs_install_pkgs_update finished
| DEBUG: Executing shell function rootfs_install_resolvconf
| DEBUG: Shell function rootfs_install_resolvconf finished
| DEBUG: Executing shell function rootfs_import_package_cache
| DEBUG: Shell function rootfs_import_package_cache finished
| DEBUG: Executing shell function rootfs_install_pkgs_download
| Reading package lists...
| Building dependency tree...
| Starting pkgProblemResolver with broken count: 5
| Starting 2 pkgProblemResolver with broken count: 5
| Investigating (0) build-essential:arm64 < none -> 12.9 @un puN Ib >
| Broken build-essential:arm64 Depends on make:arm64 < none @un pH >
|   Considering make:arm64 0 as a solution to build-essential:arm64 9999
| Investigating (0) gcc-10:arm64 < none -> 10.2.1-6 @un puN Ib >
| Broken gcc-10:arm64 Depends on cpp-10:arm64 < none @un pH > (= 10.2.1-6)
|   Considering cpp-10:arm64 -1 as a solution to gcc-10:arm64 3
| Broken gcc-10:arm64 Depends on binutils:arm64 < none @un pH > (>= 2.35.1)
|   Considering binutils:arm64 -1 as a solution to gcc-10:arm64 3
| Investigating (0) gcc:arm64 < none -> 4:10.2.1-1 @un puN Ib >
| Broken gcc:arm64 Depends on cpp:arm64 < none @un pH > (= 4:10.2.1-1)
|   Considering cpp:arm64 -1 as a solution to gcc:arm64 1
| Investigating (0) libtool:amd64 < none -> 2.4.6-15 @un puN Ib >
| Broken libtool:amd64 Depends on gcc:amd64 < none @un pH >
|   Considering gcc:amd64 1 as a solution to libtool:amd64 0
| Broken libtool:amd64 Depends on c-compiler:amd64 < none @un H >
|   Considering gcc-10:amd64 -1 as a solution to libtool:amd64 0
|   Considering gcc:amd64 1 as a solution to libtool:amd64 0
| Investigating (0) g++:arm64 < none -> 4:10.2.1-1 @un puN Ib >
| Broken g++:arm64 Depends on cpp:arm64 < none @un pH > (= 4:10.2.1-1)
|   Considering cpp:arm64 -1 as a solution to g++:arm64 0
| Done
| Some packages could not be installed. This may mean that you have
| requested an impossible situation or if you are using the unstable
| distribution that some required packages have not yet been created
| or been moved out of Incoming.
| The following information may help to resolve the situation:
| 
| The following packages have unmet dependencies:
|  build-essential:arm64 : Depends: make:arm64 but it is not installable
|  g++:arm64 : Depends: cpp:arm64 (= 4:10.2.1-1) but it is not installable
|  gcc:arm64 : Depends: cpp:arm64 (= 4:10.2.1-1) but it is not installable
|  gcc-10:arm64 : Depends: cpp-10:arm64 (= 10.2.1-6) but it is not installable
|                 Depends: binutils:arm64 (>= 2.35.1) but it is not installable
|  libtool : Depends: gcc but it is not installable or
|                     c-compiler
| E: Unable to correct problems, you have held broken packages.
| WARNING: exit code 100 from a shell command.
| DEBUG: Python function do_rootfs_install finished
| ERROR: ExecutionError('/build/tmp/work/iot2050-debian-arm64/sbuild-chroot-host/1.0-r0/temp/run.rootfs_install_pkgs_download.23820', 100, None, None)
ERROR: Task (/build/../work/isar/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb:do_rootfs_install) failed with exit code '1'
NOTE: Tasks Summary: Attempted 213 tasks of which 203 didn't need to be rerun and 1 failed.

How I understand this is that some packages (e.g. build-essential) require the installation of additional packages (make, compilers) that do not seem to be present in the offline repository cache. But my understanding of the whole build process is very far from complete, so I'd appreciate any suggestion how to fix this (although I'm well aware that an offline buils isn't really in the scope of the meta-iot2050 project).

[jan-kiszka]

Might be a regression that sneaked into Isar after its v0.9 release (via schroot support). I assume you tried meta-iot2050 master here, right? I can confirm the problem for that revision. Maybe stable/V01.03 still works (need to test, just an idea). We will have a closer look.

[jan-kiszka]

Yes, the stable, pre-sbuild branch still works fine. I will report upstream and suggest a fix once we have feedback.

[SeyerleinRichard]

Yes, I did indeed work on the master. I tried stable/V01.03 on my side, but so far I haven't been successful.

When trying to build meta-iot2050, I get some errors, the first one is as follows:

ERROR: python-gast-0.5.2-2 do_fetch: Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY) but access requested with command git -c core.fsyncobjectfiles=0 ls-remote https://salsa.debian.org/python-team/packages/python-gast.git  (for url https://salsa.debian.org/python-team/packages/python-gast.git)

When trying to build my own image (that is based on the meta-iot2050 base image) and include commit ffc3caf77d64ef67cdf58839e50cf81a41c93ab7 (last commit in stable/V01.03), I get:

ERROR: isar-bootstrap-host-1.0-r0 do_bootstrap: Execution of '/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-host/1.0-r0/temp/run.do_bootstrap.6251' failed with exit code 1:
W: qemu-debootstrap is deprecated. Please use regular debootstrap directly
I: Running command: debootstrap --verbose --variant=minbase --include=locales --no-check-gpg --components=main,contrib,non-free bullseye /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-host/1.0-r0/rootfs file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian
I: Retrieving InRelease
I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian...
E: Couldn't find these debs: apt

I'm not sure if this information is any value here, possibly the way to go would be to fix the master branch.

[jan-kiszka]

Regarding master, we may have a fix in sight, see https://groups.google.com/g/isar-users/c/TSn8O9vHJRA/m/BLvugdKBBAAJ.

Regarding the other issue, I have to admit testing only without Coral support enabled. Need to check that as well. Without Coral, the rebuild succeeded here.

[jan-kiszka]

The second issue is understood as well: Some recipes in meta-coral use tags in SRCREV, and that does not go well with offline mode (it also does not go well with upstream removing tags as it happened before with python-absl).

We need to resolve it in meta-coral, then import that into master. Is a fix on the head of the tree sufficient?

[jan-kiszka]

First step: https://github.com/siemens/meta-coral/pull/32

[SeyerleinRichard]

I can confirm that the meta-iot2050 build without Coral support works on my side.

I don't have any strong opinion on how or where to fix it as I do not use that feature in my own project.

[jan-kiszka]

Then we will fix it for the next release only for now.

[SeyerleinRichard]

The offline build is running on my side, for my image as well as for a build of meta-2050 without Coral support. I guess this is due to changes in Isar in mid-December.

I can't say anything about Coral though as I cannot build the example image with that feature (offline or online), plus I don't need it for my project.

[jan-kiszka]

Was addressed via https://github.com/siemens/meta-iot2050/pull/396, forgot to link.

Coral works in CI, but you are now the second one who says it does not build. Checking ATM.