AuroraBoot missing dependencies when running binary, not starting pixieboot when running in container.

[tbrasser]

Kairos version: quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1

CPU architecture, OS, and Version: Linux spacebar 6.4.10-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 11 Aug 2023 11:03:36 +0000 x86_64 GNU/Linux

Describe the bug When I run AuroraBoot in a container, pixieboot does not seem to start. When I run AuroraBoot directly it only works with github artifacts not with container images, then it is missing dependencies to build/serve netboot artifacts.

To Reproduce

• Running AuroraBoot container

  sudo podman run --privileged --net host --rm -i -t -v $PWD/aurora.yaml:/aurora.yaml -v $PWD/build:/tmp/auroraboot -v $PWD/data:/tmp/data quay.io/kairos/auroraboot --debug /aurora.yaml

• Running AuroraBoot binary

  sudo ./AuroraBoot --debug ./aurora.yaml

Expected behavior When I use AuroraBoot binary, it should be able to build/serve iso/netboot artifacts. When I use AuroraBoot in a container with host networking, it should have no problems serving all it needs to.

Logs

• Running binary:

  5:07PM INF Pulling container image 'quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1' to '/home/thomas/kairos/build/temp-rootfs' (local: true)
  5:08PM DBG Output 'INFO   Downloading quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1 to /home/thomas/kairos/build/temp-rootfs
  INFO   Size: 2.402GiB'
  5:08PM INF Generating iso 'kairos' from '/home/thomas/kairos/build/temp-rootfs' to '/home/thomas/kairos/build/iso'
  5:08PM DBG Output '/bin/sh: line 1: /entrypoint.sh: No such file or directory'
  5:08PM ERR Failed generating iso 'kairos' from '/home/thomas/kairos/build/temp-rootfs'. Error: exit status 127
  4 errors occurred:
  * exit status 127
  * 'extract-netboot' deps gen-iso failed
  * 'start-httpserver' deps gen-iso failed
  * 'start-netboot' deps extract-netboot failed

• Running in container:

  
  3:33PM DBG #cloud-config
  install:
  auto: true
  device: /dev/nvme1n1
  reboot: true
  hostname: kairos-node-{{ trunc 4 .MachineID }}
  users:
  - name: kairos
  passwd: kairos
  ssh_authorized_keys:
    - github:tbrasser
  debug: true
  strict: true
  k3s:
  enable: true
  bundles:
  - targets:
    - run://quay.io/kairos/community-bundles:system-upgrade-controller_latest
    - run://quay.io/kairos/community-bundles:cert-manager_latest
    - run://quay.io/kairos/community-bundles:kairos_latest
  kairos:
  entangle:
  enable: true
  entangleProxy:
  enable: true
  p2p:
  role: master
  network_token: |
  <redacted>
  reset:
  reboot: true
  reset-persistent: true
  reset-oem: true
  upgrade:
  reboot: true

3:33PM DBG 1. 3:33PM DBG (background: false) 3:33PM DBG (background: false) 3:33PM DBG (background: false) 3:33PM DBG 3:33PM DBG 2. 3:33PM DBG (background: false) 3:33PM DBG (background: false) 3:33PM DBG 3:33PM DBG 3. 3:33PM DBG (background: false) 3:33PM DBG 3:33PM DBG 4. 3:33PM DBG (background: true) 3:33PM DBG (background: false) 3:33PM DBG 3:33PM DBG 5. 3:33PM DBG (background: true) 3:33PM DBG 3:33PM INF Pulling container image 'quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1' to '/home/thomas/kairos/build/temp-rootfs' (local: false) 3:37PM DBG Output ' INFO Downloading quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1 to /home/thomas/kairos/build/temp-rootfs INFO Pulled: sha256:e01b4602b73d840b452dcfe3750d9e331b11adea3baaaa11be81f5dd8afe6239 quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1 INFO Size: 2.402GiB ' 3:37PM INF Generating iso 'kairos' from '/home/thomas/kairos/build/temp-rootfs' to '/home/thomas/kairos/build/iso' 3:40PM DBG Output '+ enki --config-dir /config --debug --name kairos build-iso --squash-no-compression --overlay-iso /tmp/geniso69068545 --date=false --output /home/thomas/kairos/build/iso dir:/home/thomas/kairos/build/temp-rootfs INFO[2023-08-22T15:37:57Z] Starting enki version 0.0.1
DEBU[2023-08-22T15:37:57Z] {Version:0.0.1 GitCommit: GoVersion:go1.20.6} DEBU[2023-08-22T15:37:57Z] Full config loaded: &v1.BuildConfig{ Date: false, Name: "kairos", OutDir: "/home/thomas/kairos/build/iso", Config: v1.Config{ Logger: &v1.logrusWrapper{ // p0 Logger: &logrus.Logger{ Out: &os.File{}, Hooks: logrus.LevelHooks{}, Formatter: &logrus.TextFormatter{ ForceColors: true, DisableColors: false, ForceQuote: false, DisableQuote: false, EnvironmentOverrideColors: false, DisableTimestamp: false, FullTimestamp: true, TimestampFormat: "", DisableSorting: false, SortingFunc: , DisableLevelTruncation: false, PadLevelText: false, QuoteEmptyFields: false, FieldMap: logrus.FieldMap(nil), CallerPrettyfier: , }, ReportCaller: false, Level: 5, ExitFunc: os.Exit, BufferPool: nil, }, }, Fs: &vfs.osfs{}, // p1 Mounter: &mount.Mounter{}, Runner: &v1.RealRunner{ // p2 Logger: p0, }, Syscall: &v1.RealSyscall{}, CloudInitRunner: &cloudinit.YipCloudInitRunner{}, ImageExtractor: nil, Client: &http.Client{}, Platform: &v1.Platform{ OS: "linux", Arch: "x86_64", GolangArch: "amd64", }, Cosign: false, Verify: false, CosignPubKey: "", Repos: []v1.Repository{}, Arch: "x86_64", SquashFsCompressionConfig: []string{}, SquashFsNoCompression: true, }, } DEBU[2023-08-22T15:37:57Z] Loaded LiveISO: &v1.LiveISO{ RootFS: nil, UEFI: []v1.ImageSource{ &v1.ImageSource{}, }, Image: []v1.ImageSource{ &v1.ImageSource{}, &v1.ImageSource{}, }, Label: "COS_LIVE", GrubEntry: "Kairos", BootloaderInRootFs: false, } INFO[2023-08-22T15:37:57Z] Preparing squashfs root...
INFO[2023-08-22T15:37:57Z] Copying /home/thomas/kairos/build/temp-rootfs source... INFO[2023-08-22T15:37:57Z] Starting rsync...
DEBU[2023-08-22T15:37:57Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /home/thomas/kairos/build/temp-rootfs/ /tmp/enki-iso324095657/rootfs/' DEBU[2023-08-22T15:38:02Z] Syncing data...
DEBU[2023-08-22T15:38:07Z] Syncing data...
INFO[2023-08-22T15:38:07Z] Finished syncing
INFO[2023-08-22T15:38:07Z] Finished copying /home/thomas/kairos/build/temp-rootfs into /tmp/enki-iso324095657/rootfs INFO[2023-08-22T15:38:07Z] Preparing EFI image...
INFO[2023-08-22T15:38:07Z] Copying /efi source...
INFO[2023-08-22T15:38:07Z] Starting rsync...
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /efi/ /tmp/enki-iso324095657/uefi/' INFO[2023-08-22T15:38:07Z] Finished syncing
INFO[2023-08-22T15:38:07Z] Finished copying /efi into /tmp/enki-iso324095657/uefi INFO[2023-08-22T15:38:07Z] Preparing ISO image root tree...
INFO[2023-08-22T15:38:07Z] Copying /efi source...
INFO[2023-08-22T15:38:07Z] Starting rsync...
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /efi/ /tmp/enki-iso324095657/iso/' INFO[2023-08-22T15:38:07Z] Finished syncing
INFO[2023-08-22T15:38:07Z] Finished copying /efi into /tmp/enki-iso324095657/iso INFO[2023-08-22T15:38:07Z] Copying /grub2 source...
INFO[2023-08-22T15:38:07Z] Starting rsync...
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /grub2/ /tmp/enki-iso324095657/iso/' INFO[2023-08-22T15:38:07Z] Finished syncing
INFO[2023-08-22T15:38:07Z] Finished copying /grub2 into /tmp/enki-iso324095657/iso INFO[2023-08-22T15:38:07Z] Copying /tmp/geniso69068545 source...
INFO[2023-08-22T15:38:07Z] Starting rsync...
DEBU[2023-08-22T15:38:07Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/mnt --exclude=/proc --exclude=/sys --exclude=/dev --exclude=/tmp --exclude=/host --exclude=/run /tmp/geniso69068545/ /tmp/enki-iso324095657/iso/' INFO[2023-08-22T15:38:07Z] Finished syncing
INFO[2023-08-22T15:38:07Z] Finished copying /tmp/geniso69068545 into /tmp/enki-iso324095657/iso DEBU[2023-08-22T15:38:07Z] Copying Kernel file /tmp/enki-iso324095657/rootfs/boot/vmlinuz-6.2.0-27-generic to iso root tree DEBU[2023-08-22T15:38:07Z] Copying initrd file /tmp/enki-iso324095657/rootfs/boot/initrd-6.2.0-27-generic to iso root tree INFO[2023-08-22T15:38:07Z] Creating squashfs...
DEBU[2023-08-22T15:38:07Z] Running cmd: 'mksquashfs /tmp/enki-iso324095657/rootfs /tmp/enki-iso324095657/iso/rootfs.squashfs -b 1024k' INFO[2023-08-22T15:40:29Z] Creating EFI image...
INFO[2023-08-22T15:40:29Z] Creating file system image /tmp/enki-iso324095657/iso/boot/uefi.img DEBU[2023-08-22T15:40:29Z] Running cmd: 'mkfs.vfat -n COS_GRUB /tmp/enki-iso324095657/iso/boot/uefi.img' DEBU[2023-08-22T15:40:29Z] Running cmd: 'mcopy -s -i /tmp/enki-iso324095657/iso/boot/uefi.img /tmp/enki-iso324095657/uefi/EFI ::' DEBU[2023-08-22T15:40:29Z] Running cmd: 'mcopy -s -i /tmp/enki-iso324095657/iso/boot/uefi.img /tmp/enki-iso324095657/uefi/var ::' INFO[2023-08-22T15:40:29Z] Creating ISO image...
DEBU[2023-08-22T15:40:29Z] Running cmd: 'xorriso -volid COS_LIVE -joliet on -padding 0 -outdev /home/thomas/kairos/build/iso/kairos.iso -map /tmp/enki-iso324095657/iso / -chmod 0755 -- -boot_image grub bin_path=/boot/x86_64/loader/eltorito.img -boot_image grub grub2_mbr=/tmp/enki-iso324095657/iso//boot/x86_64/loader/boot_hybrid.img -boot_image grub grub2_boot_info=on -boot_image any partition_offset=16 -boot_image any cat_path=/boot/x86_64/boot.catalog -boot_image any cat_hidden=on -boot_image any boot_info_table=on -boot_image any platform_id=0x00 -boot_image any emul_type=no_emulation -boot_image any load_size=2048 -append_partition 2 0xef /tmp/enki-iso324095657/iso/boot/uefi.img -boot_image any next -boot_image any efi_path=--interval:appended_partition_2:all:: -boot_image any platform_id=0xef -boot_image any emul_type=no_emulation' DEBU[2023-08-22T15:40:32Z] Xorriso: xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev '/home/thomas/kairos/build/iso/kairos.iso' Media current: stdio file, overwriteable Media status : is blank Media summary: 0 sessions, 0 data blocks, 0 data, 182g free xorriso : UPDATE : 1168 files added in 1 seconds Added to ISO image: directory '/'='/tmp/enki-iso324095657/iso' xorriso : NOTE : Copying to System Area: 512 bytes from file '/tmp/enki-iso324095657/iso/boot/x86_64/loader/boot_hybrid.img' xorriso : UPDATE : Writing: 12896s 2.0% fifo 9% buf 50% xorriso : UPDATE : Writing: 64528s 10.2% fifo 41% buf 50% xorriso : UPDATE : Writing: 566736s 89.3% fifo 99% buf 50% 741.9xD ISO image produced: 634343 sectors Written to medium : 634352 sectors at LBA 48 Writing to '/home/thomas/kairos/build/iso/kairos.iso' completed successfully.

' 3:40PM INF Extracting netboot artifacts 'kairos' from '/home/thomas/kairos/build/iso/kairos.iso' to '/home/thomas/kairos/build/netboot' 3:40PM INF Listening on 0.0.0.0:8080... 3:40PM DBG Output '' 3:40PM INF Start pixiecore

[tbrasser]

Tried building with osbuilder on kind and getting better results (@2.4.0-rc1)!

netboot: true works, but gkeImage: true doesn't. At least I got a WoW now that is usable.

Can update here or close and create new issues when they come up.

[audiato]

I'm getting the same errors when trying to use ghcr.io/kairos-io/provider-rke2:v2.3.3 with Auroraboot in Docker.

https://github.com/kairos-io/kairos/discussions/1793

[jimmykarily]

@tbrasser I don't have podman installed so I just replaced podman with docker in your command:

sudo docker run --privileged --net host --rm -i -t -v $PWD/aurora.yaml:/aurora.yaml -v $PWD/build:/tmp/auroraboot -v $PWD/data:/tmp/data quay.io/kairos/auroraboot --debug /aurora.yaml

also I don't know what your aurora.yaml looks like. I created one that would pull the same image as in the description:

container_image: quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1

It works just fine. Among the rest of the output, I see this line:

6:52AM INF Pulling container image 'quay.io/kairos/kairos-ubuntu:v2.4.0-alpha2-k3sv1.27.3-k3s1' to '/tmp/temp-rootfs' (local: false)

Your output says local: true. Are you prefixing your image in the aurora config with docker:// ? This will make auroraboot try to use a local image and not pull it.

[tbrasser]

I tried various combinarions, will try again with 2.4.0 release. What is the suggested WoW? osbuilder or auroraboot? (also encountering some race conditions in the main/sidecar containers there, seems to me all should be initContainers with exporter to nginx as single main container. (also probably job instead of bare pod))

[jimmykarily]

Regarding osbuilder, I agree. On osbuilder master branch, we've already switched to InitContainers (not tagged or released yet). Also, it used to be a Job but was later converted to a Pod. I still think it should be a Job and we'll probably change it back to a Job in the future.

There is a lot of work planned on osbuilder and when we are done, auroraboot will be used internally by osbuilder but will also be used directly by users that don't want to use Kubernetes for building.

Relevant issue with more information: https://github.com/kairos-io/kairos/issues/1633

[jimmykarily]

@tbrasser since we can't reproduce it, I'll close this. Please give it a try with 2.4.0 and feel free to re-open is the issue persists. Thanks for reporting!