Build of live-image-arm64.hybrid.iso failed.

[djkaosun]

Hi,

apparently building live-image-arm64.hybrid.iso seem to fail.

I ran the following command:

$ sudo docker run --rm -it --platform arm64 --privileged \
        -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static \
        -v "$(pwd)":/vyos \
        -v /dev:/dev \
        --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash \
        -c 'cd /vyos; /bin/bash -x build-image.sh'

The result is as follows:

Current UID/GID: 0/0
+ set -x
+ set -e
++ pwd
+ ROOTDIR=/vyos
+ rm -rf vyos-build
+ git clone http://github.com/vyos/vyos-build vyos-build
Cloning into 'vyos-build'...
warning: redirecting to https://github.com/vyos/vyos-build/
remote: Enumerating objects: 11133, done.
remote: Counting objects: 100% (589/589), done.
remote: Compressing objects: 100% (280/280), done.
remote: Total 11133 (delta 309), reused 472 (delta 275), pack-reused 10544
Receiving objects: 100% (11133/11133), 7.39 MiB | 9.92 MiB/s, done.
Resolving deltas: 100% (6602/6602), done.
++ find build -type f -name '*.deb'
++ grep -v -e -dbgsym_ -e libnetfilter-conntrack3-dbg
find: ‘build’: No such file or directory
+ cd vyos-build
+ echo 'Copy new default configuration to the vyos image'
Copy new default configuration to the vyos image
+ cp /vyos/config.boot.default data/live-build-config/includes.chroot/opt/vyatta/etc/config.boot.default
+ VYOS_BUILD_FLAVOR='data/generic-arm64.json ./configure'
+ make iso
[2022-12-15 12:38:32] lb clean 
P: Cleaning chroot
set -o pipefail
./build-vyos-image iso
exit 0
I: Checking if packages required for VyOS image build are installed
build/config
I: Cleaning the build workspace
[2022-12-15 12:38:37] lb clean 
P: Cleaning chroot
I: Setting up additional APT entries
I: Configuring live-build
[2022-12-15 12:38:39] lb config noauto --architectures amd64 --bootappend-live boot=live components hostname=vyos username=live nopersistence noautologin nonetworking union=overlay console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 --bootappend-live-failsafe live components memtest noapic noapm nodma nomce nolapic nomodeset nosmp nosplash vga=normal console=ttyS0,115200 console=tty0 net.ifnames=0 biosdevname=0 --linux-flavours amd64-vyos --linux-packages linux-image-5.15.82 --bootloader syslinux,grub-efi --binary-images iso-hybrid --checksums sha256 md5 --debian-installer none --distribution bullseye --iso-application VyOS --iso-publisher root@602803ccb2b2 --iso-volume VyOS --debootstrap-options --variant=minbase --exclude=isc-dhcp-client,isc-dhcp-common,ifupdown --include=apt-utils,ca-certificates,gnupg2 --mirror-bootstrap http://deb.debian.org/debian --mirror-chroot http://deb.debian.org/debian --mirror-chroot-security http://deb.debian.org/debian-security --mirror-binary http://deb.debian.org/debian --mirror-binary-security http://deb.debian.org/debian-security --archive-areas main contrib non-free --firmware-chroot false --firmware-binary false --updates true --security true --backports true --apt-recommends false --apt-options --yes -oAPT::Get::allow-downgrades=true --apt-indices false
P: Updating config tree for a debian/bullseye/amd64 system
P: Symlinking hooks...
I: Starting image build
[2022-12-15 12:38:43] lb build 
P: live-build 20210407
P: Building for a debian/bullseye/amd64 system
[2022-12-15 12:38:44] lb bootstrap 
P: Setting up clean exit handler
[2022-12-15 12:38:45] lb bootstrap_cache restore
W: skipping /usr/lib/live/build/bootstrap_cache, foreign architecture(s).
[2022-12-15 12:38:46] lb bootstrap_debootstrap 
W: skipping /usr/lib/live/build/bootstrap_debootstrap, foreign architecture(s).
[2022-12-15 12:38:47] lb bootstrap_cache save
W: skipping /usr/lib/live/build/bootstrap_cache, foreign architecture(s).
[2022-12-15 12:38:48] lb chroot_prep install all-except-archives
E: the following stage is required to be done first: bootstrap
E: An unexpected failure occurred, exiting...

What am I doing wrong?

[djkaosun]

I posted this on the VyOS forum, as it is probably a problem with the VyOS build system.

https://forum.vyos.io/t/build-for-arm64-fails/10018

[djkaosun]

I am not sure if this is the correct solution, but I changed build-image.sh of vyos-pi-builder as follows, and the build was completed tentatively. I have not yet confirmed that the VyOS image built with this script works properly.

build-image.sh is:

- # Clean out the build-repo and copy all custom packages
- rm -rf vyos-build
- git clone http://github.com/vyos/vyos-build vyos-build
  for a in $(find build -type f -name "*.deb" | grep -v -e "-dbgsym_" -e "libnetfilter-conntrack3-dbg"); do
    echo "Copying package: $a"
    cp $a vyos-build/packages/
  done

  cd vyos-build

  echo "Copy new default configuration to the vyos image"
  cp ${ROOTDIR}/config.boot.default data/live-build-config/includes.chroot/opt/vyatta/etc/config.boot.default

  # Build the image
- VYOS_BUILD_FLAVOR=data/generic-arm64.json
- ./configure
- make iso
+ /vyos/vyos-build/build-vyos-image iso --architecture arm64

  cd $ROOTDIR

  # Build u-boot
  bash build-u-boot.sh

Makefile is:

  container:
-   sudo rm -rf vyos-build
-   git clone -b current --single-branch https://github.com/vyos/vyos-build
    sudo docker build --arch arm64 vyos-build/docker -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -t vyos/vyos-build:current-arm64

- iso-local:
-   sudo docker run --rm -it --arch arm64 --privileged -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -v "$(shell pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 localhost/vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-image.sh'

  iso-registry:
-   sudo docker run --rm -it --arch arm64 --privileged -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -v "$(shell pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-image.sh'
+   sudo docker run --rm -it --platform arm64 --privileged -v "$(shell pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-image.sh'

In addition, the following preparations were required.

$ git clone https://github.com/runborg/vyos-pi-builder
$ cd vyos-pi-builder
$ git clone -b current --single-branch https://github.com/vyos/vyos-build

$ wget -qO- https://repos.influxdata.com/influxdb.key > sudo tee vyos-build/data/live-build-config/archives/influxdb.key.chroot

$ vi vyos-build/data/architectures/arm64.toml

+ additional_repositories = [
+   "deb [arch=arm64] https://repos.influxdata.com/debian bullseye stable"
+ ]

+ kernel_flavor = "arm64-vyos"

  # Packages added to images for x86 by default
  packages = [
-   "grub-efi-arm"
+   "grub-efi-arm64",
+   "telegraf"
  ]
  bootloaders = "grub-efi"

$ vi build-pi-kernel.sh

set -x
set -e
ROOTDIR=$(pwd)
cd vyos-build/packages/linux-kernel/
git clone https://github.com/raspberrypi/linux
./build-kernel.sh
git clone https://github.com/accel-ppp/accel-ppp.git
./build-accel-ppp.sh

$sudo docker run --rm -it --platform arm64 --privileged -v "$(pwd)":/vyos -v /dev:/dev --sysctl net.ipv6.conf.lo.disable_ipv6=0 vyos/vyos-build:current-arm64 /bin/bash -c 'cd /vyos; /bin/bash -x build-pi-kernel.sh'

[djkaosun]

I wanted to use wi-fi but it is not recognizing the device and I am trying to work it out.

[djkaosun]

I forgot to build the firmware, so I did the following additionally.

build-pi-kernel.sh:

  set -x
  set -e
  ROOTDIR=$(pwd)
  cd vyos-build/packages/linux-kernel/
  git clone https://github.com/raspberrypi/linux
  ./build-kernel.sh
+ ln -s linux/drivers/firmware linux-firmware
+ ./build-linux-firmware.sh
  git clone https://github.com/accel-ppp/accel-ppp.git
  ./build-accel-ppp.sh

vyos-build/data/architectures/arm64.toml:

  additional_repositories = [
    "deb [arch=arm64] https://repos.influxdata.com/debian bullseye stable"
  ]

  kernel_flavor = "arm64-vyos"

  # Packages added to images for x86 by default
  packages = [
    "grub-efi-arm64",
+   "vyos-linux-firmware",
    "telegraf"
  ]
  bootloaders = "grub-efi"

The wi-fi device did not work even after running build-linux-firmware.sh. However, I did the following on VyOS and it is now recognized.

$ sudo -i
# wget -qO- https://github.com/openwrt/cypress-nvram/blob/master/brcmfmac43456-sdio.txt > /lib/firmware/brcm/brcmfmac43455-sdio.txt

In addition, because the kernel version included the unfamiliar "v8", the following workaround was implemented.

vyos-build/build-vyos-image:

- --linux-packages linux-image-{{kernel_version}} \
+ --linux-packages linux-image-{{kernel_version}}-v8 \

vyos-build/packages/linux-kernel/build-linux-firmware.sh:

- FW_FILES=$(find ${LINUX_SRC}/debian/linux-image/lib/modules/${KERNEL_VERSION}${KERNEL_SUFFIX}/kernel/drivers/net -name *.ko | xargs modinfo | grep "^firmware:" | awk '{print $2}')
+ FW_FILES=$(find ${LINUX_SRC}/debian/linux-image/lib/modules/${KERNEL_VERSION}-v8${KERNEL_SUFFIX}/kernel/drivers/net -name *.ko | xargs modinfo | grep "^firmware:" | awk '{print $2}')

It appears to be working at least for the moment, so I will use this image for a while.

[runborg]

As for now all fixes should be merged, i’ve verified that it builds fine, but do not have any means to verify it.. could you do a verification?