Open bobmorane83 opened 4 months ago
Hi @bobmorane83!
There is a separate make target for building an RPI disk.
Try running 'make ARCH=arm64 build build-os build-rpi-disk' and it should build the toolkit, green OS container image and finally a raw image that can be written to an sd-card and booted!
Let us know how it goes! 👍
Thanks @frelon, Now I got :
Building arm64 disk mkdir -p /home/pi/Developement/elemental/elemental-toolkit/build docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /home/pi/Developement/elemental/elemental-toolkit/examples:/examples -v /home/pi/Developement/elemental/elemental-toolkit/build:/build \ --entrypoint /usr/bin/elemental \ local/elemental-toolkit:v2.1.0-dev-gc608e0b08 --debug build-disk --platform linux/arm64 --cloud-init-paths /examples/green --unprivileged --expandable -n elemental-green.aarch64 --local \ --squash-no-compression --deploy-command elemental,--debug,reset,--reboot,--disable-boot-entry -o /build local/elemental-green:v2.1.0-dev-gc608e0b08 Error: unknown flag: --unprivileged
Removing the --unprivileged flag I got :
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /home/pi/Developement/elemental/elemental-toolkit/examples:/examples -v /home/pi/Developement/elemental/elemental-toolkit/build:/build \ --entrypoint /usr/bin/elemental \ local/elemental-toolkit:v2.1.0-dev-gc608e0b08 --debug build-disk --platform linux/arm64 --cloud-init-paths /examples/green --expandable -n elemental-green.aarch64 --local \ --squash-no-compression --deploy-command elemental,--debug,reset,--reboot,--disable-boot-entry -o /build local/elemental-green:v2.1.0-dev-gc608e0b08 Error: accepts 0 arg(s), received 1
I also tried with tag v2.1.0 but same errors.
Any comment ? Thx,
Ok, if i use FLAVOR to green-rpi, remove --unprivileged and add --system, and make build-rpi-disk, I got a raw file, flashed on SD card and get the following ;
-Boot 2021.10 (Aug 04 2022 - 12:00:00 +0000)
DRAM: 2 GiB RPI 4 Model B (0xb03114) MMC: mmcnr@7e300000: 1, mmc@7e340000: 0 Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In: serial Booting `Elemental recovery'
U-Boot 2021.10 (Aug 04 2022 - 12:00:00 +0000)
DRAM: 2 GiB RPI 4 Model B (0xb03114) Booting `Elemental'
error: ../../grub-core/commands/search.c:296:no such device: COS_STATE. error: ../../grub-core/fs/fshelp.c:258:file `/.snapshots/active' not found. error: ../../grub-core/net/net.c:1317:no server is specified. error: ../../grub-core/loader/arm64/efi/linux.c:301:you need to load the kernel first.
Press any key to continue...
Falling back to `Elemental'
error: ../../grub-core/commands/search.c:296:no such device: COS_STATE. error: ../../grub-core/fs/fshelp.c:258:file `/.snapshots/active' not found. error: ../../grub-core/net/net.c:1317:no server is specified. error: ../../grub-core/loader/arm64/efi/linux.c:301:you need to load the kernel first.
Press any key to continue...
Falling back to `Elemental recovery'
EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services...
Any doc to read to understand the next steps ?
Thanks, BR
Good catch! If you want to put up a PR with the changes I would be happy to review!
From the logs it looks like it's trying to boot the linux kernel from recovery, which is sort of the expected behavior.. when building expandable images we boot into the recovery partition to expand the partition table correctly and then we reboot into the active system. This might take a while, do you get any more output after the EFI stub: Exiting boot services...
log?
Waiting for an hour or so and nothing appends ... GRUB look like this and key in do not works :
Good news : After waiting enought :
ssh 192.168.2.194 -l pi Password:
But what password ? elemental ? nop ... in user-data I have :
users:
- name: pi
groups: users,adm,dialout,audio,netdev,video,plugdev,cdrom,games,input,gpio,spi,i2c,render,sudo
shell: /bin/bash
lock_passwd: false
passwd: $5$hJfZB7KyQ3$nZYJJcuk6mlRH/evlBMRmYzgXUrRde3WlAh9.yzYW2B
But what kind of hash ?
Awesome! The default login for the examples is root/elemental
.. But maybe root login is not enabled for rpi by default. You could probably copy the line here to green-rpi dockerfile.
Was : https://rancher.github.io/elemental-toolkit/docs/customizing/login/
❯ ssh 192.168.2.194 -l root
Password:
Last failed login: Fri May 17 12:59:57 UTC 2024 from 192.168.2.151 on ssh:notty
There were 2 failed login attempts since the last successful login.
Have a lot of fun...
elemental:~ #
Nice find! Yea we recently switched from cos to elemental as the example password.. Glad you got it working! 👍
Ok, now the RPi is booting as expecting : First in recovery, and after a delay( few minutes) rebooting in "elemental" and picking up a DHCP IP (BTW "elemental" is now the normal password as you mentionned). I have added k3s and k3s-agent to the container and I now creating a cluster with elemental node as worker and standard OS Pi 4 with k3s as master.
Everything is Ok at first :
If I reboot the worker node (elemental1), the node do not reconnect by itself, I get :
msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
The node has unique hostname and adding --with-node-id add effectively an ID but different after each boot.
I have, in the master node, to delete the worker node elemental1 to get the node back and ready.
BTW I had to add "--snapshotter=native" option to get k3s running on elemental in /etc/systemd/system/k3s-agent.service. All other option (and token) are stored in /etc/systemd/system/k3s-agent.service.env.
Any clue ? Thanks,
Very nice work!
/etc/rancher
is probably ephemeral and will be cleared during reboot, in elemental we add that dir as persistent here
So you could probably copy that file and put it in /system/oem
and it should persist the /etc/rancher/node/password
file :+1:
Effectively putting back the original password in /etc/rancher/node/password
the node connect itself right away 👍
But I didn't catch how to copy the file in the read only /system/oem
(can't remount it in rw) and say to k3s-agent to access there ?
You can put it in /oem
also, that should be writeable by default!
Adding
ephemeral:
type: tmpfs
size: 25%
paths: ['/etc', '/var', '/srv']
persistent:
mode: bind
paths:
- /etc/systemd
- /etc/rancher
- /etc/ssh
- /etc/iscsi
- /etc/cni
- /home
- /opt
- /root
- /usr/libexec
- /usr/local
- /var/log
- /var/lib/elemental
- /var/lib/rancher
- /var/lib/kubelet
- /var/lib/NetworkManager
- /var/lib/cni
- /var/lib/calico
to pkg/features/embedded/cloud-config-defaults/system/oem/00_layout.yaml
did not do the trick.
I think, I'am still have to understand elemental better ...
Any help ?
Thanks,
Okay, if you could post the entire file contents here that would help.. It looks like just adding it there as is will only run when booted to recovery because of this conditional: - if: '[ ! -f "/run/elemental/recovery_mode" ]'
I simply copy/past the part of framework/files/system/oem/01_elemental-rootfs.yaml
from elemental repo :
# Layout Elemental OEM configuration file
#
# This file is part of Elemental and will get reset during upgrades.
#
# Before you change this file manually,
# consider copying this file to /usr/local/cloud-config or
# copy the file with a prefix starting by 90, e.g. /oem/91_custom.yaml
name: "Rootfs Layout Settings"
stages:
rootfs:
- if: '[ ! -f "/run/elemental/recovery_mode" ]'
name: "Layout configuration"
files:
- path: /run/elemental/config.d/layout.yaml
content: |
mount:
write-fstab: true
ephemeral:
type: tmpfs
size: 25%
paths: ['/etc', '/var', '/srv']
persistent:
mode: bind
paths:
- /etc/systemd
- /etc/rancher
- /etc/ssh
- /etc/iscsi
- /etc/cni
- /home
- /opt
- /root
- /usr/libexec
- /usr/local
- /var/log
- /var/lib/elemental
- /var/lib/rancher
- /var/lib/kubelet
- /var/lib/NetworkManager
- /var/lib/cni
- /var/lib/calico
Another option I was thinking of is to add :
name: "Run on initramfs"
stages:
initramfs:
- name: "Rancher Setting"
if: '[ ! -f "/run/cos/recovery_mode" ]'
commands:
- |
ln -s /oem /etc/rancher
Could you post the output of mount
and journalctl -u elemental-setup-rootfs
?
elemental:~ # mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=385324k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p4 on /run/initramfs/elemental-state type ext4 (rw,relatime)
/run/initramfs/elemental-state/.snapshots/1/snapshot.img on / type ext2 (ro,relatime)
/dev/mmcblk0p2 on /oem type ext4 (rw,relatime)
/dev/mmcblk0p1 on /run/elemental/efi type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p5 on /run/elemental/persistent type ext4 (rw,relatime)
tmpfs on /run/elemental/overlay type tmpfs (rw,relatime,size=481656k,inode64)
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/elemental/overlay/etc.overlay/upper,workdir=/run/elemental/overlay/etc.overlay/work)
overlay on /var type overlay (rw,relatime,lowerdir=/sysroot/var,upperdir=/run/elemental/overlay/var.overlay/upper,workdir=/run/elemental/overlay/var.overlay/work)
overlay on /srv type overlay (rw,relatime,lowerdir=/sysroot/srv,upperdir=/run/elemental/overlay/srv.overlay/upper,workdir=/run/elemental/overlay/srv.overlay/work)
/dev/mmcblk0p5 on /home type ext4 (rw,relatime)
/dev/mmcblk0p5 on /root type ext4 (rw,relatime)
/dev/mmcblk0p5 on /opt type ext4 (rw,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18473)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=192660k,nr_inodes=48165,mode=700,inode64)
elemental:~ # journalctl -u elemental-setup-rootfs
Jan 08 12:00:24 recovery.lan systemd[1]: Starting Elemental system early rootfs setup...
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Starting elemental version v2.1.0
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Reading configuration from '/etc/elemental'
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Running stage: rootfs.before
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
Jan 08 12:00:24 recovery.lan elemental[662]: WARN[2024-01-08T12:00:24Z] (conditional) Skip 'Skipping stage (if statement error: failed to run cat /proc/cmdline | grep -q "CDLABEL" || cat /proc/cmdline | grep -q "elemental.disable" || cat /proc/cmdline | grep -q "rd.cos.disable"
Jan 08 12:00:24 recovery.lan elemental[662]: : exit status 1)' stage name: Identify live mode
Jan 08 12:00:25 recovery.lan elemental[662]: ERRO[2024-01-08T12:00:25Z] no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[662]: WARN[2024-01-08T12:00:25Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/elemental/live_mode" ]: exit status 1)' stage name: Include cloud init files from the ISO root
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Processing stage step 'Layout configuration'. ( commands: 0, files: 1, ... )
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.before
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
Jan 08 12:00:25 recovery.lan elemental[662]: WARN[2024-01-08T12:00:25Z] 1 error occurred:
Jan 08 12:00:25 recovery.lan elemental[662]: * no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[662]:
Jan 08 12:00:25 recovery.lan systemd[1]: Finished Elemental system early rootfs setup.
Hmm, looks like the default configuration on the recovery.. does it look the same when booted to the active snapshot?
Sound the same ...
elemental:~ # mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=385324k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p4 on /run/initramfs/elemental-state type ext4 (rw,relatime)
/run/initramfs/elemental-state/.snapshots/1/snapshot.img on / type ext2 (ro,relatime)
/dev/mmcblk0p2 on /oem type ext4 (rw,relatime)
/dev/mmcblk0p1 on /run/elemental/efi type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p5 on /run/elemental/persistent type ext4 (rw,relatime)
tmpfs on /run/elemental/overlay type tmpfs (rw,relatime,size=481656k,inode64)
overlay on /var type overlay (rw,relatime,lowerdir=/sysroot/var,upperdir=/run/elemental/overlay/var.overlay/upper,workdir=/run/elemental/overlay/var.overlay/work)
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/elemental/overlay/etc.overlay/upper,workdir=/run/elemental/overlay/etc.overlay/work)
overlay on /srv type overlay (rw,relatime,lowerdir=/sysroot/srv,upperdir=/run/elemental/overlay/srv.overlay/upper,workdir=/run/elemental/overlay/srv.overlay/work)
overlay on /home type overlay (rw,relatime,lowerdir=/sysroot/home,upperdir=/run/elemental/persistent/.state/home.overlay/upper,workdir=/run/elemental/persistent/.state/home.overlay/work)
overlay on /opt type overlay (rw,relatime,lowerdir=/sysroot/opt,upperdir=/run/elemental/persistent/.state/opt.overlay/upper,workdir=/run/elemental/persistent/.state/opt.overlay/work)
overlay on /root type overlay (rw,relatime,lowerdir=/sysroot/root,upperdir=/run/elemental/persistent/.state/root.overlay/upper,workdir=/run/elemental/persistent/.state/root.overlay/work)
overlay on /etc/systemd type overlay (rw,relatime,lowerdir=/sysroot/etc/systemd,upperdir=/run/elemental/persistent/.state/etc-systemd.overlay/upper,workdir=/run/elemental/persistent/.state/etc-systemd.overlay/work)
overlay on /etc/ssh type overlay (rw,relatime,lowerdir=/sysroot/etc/ssh,upperdir=/run/elemental/persistent/.state/etc-ssh.overlay/upper,workdir=/run/elemental/persistent/.state/etc-ssh.overlay/work)
overlay on /var/log type overlay (rw,relatime,lowerdir=/sysroot/var/log,upperdir=/run/elemental/persistent/.state/var-log.overlay/upper,workdir=/run/elemental/persistent/.state/var-log.overlay/work)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18970)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=963312k,nr_inodes=1048576,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=192660k,nr_inodes=48165,mode=700,inode64)
elemental:~ # journalctl -u elemental-setup-rootfs
Jan 08 12:00:24 recovery.lan systemd[1]: Starting Elemental system early rootfs setup...
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Starting elemental version v2.1.0
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Reading configuration from '/etc/elemental'
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Running stage: rootfs.before
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
Jan 08 12:00:24 recovery.lan elemental[661]: WARN[2024-01-08T12:00:24Z] (conditional) Skip 'Skipping stage (if statement error: failed to run cat /proc/cmdline | grep -q "CDLABEL" || cat /proc/cmdline | grep -q "elemental.disable" || cat /proc/cmdline | grep -q "rd.cos.disable"
Jan 08 12:00:24 recovery.lan elemental[661]: : exit status 1)' stage name: Identify live mode
Jan 08 12:00:25 recovery.lan elemental[661]: ERRO[2024-01-08T12:00:25Z] no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[661]: WARN[2024-01-08T12:00:25Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/elemental/live_mode" ]: exit status 1)' stage name: Include cloud init files from the ISO root
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Processing stage step 'Layout configuration'. ( commands: 0, files: 1, ... )
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.before
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
Jan 08 12:00:25 recovery.lan elemental[661]: WARN[2024-01-08T12:00:25Z] 1 error occurred:
Jan 08 12:00:25 recovery.lan elemental[661]: * no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[661]:
Jan 08 12:00:25 recovery.lan systemd[1]: Finished Elemental system early rootfs setup.
Can you post result of elemental version
as well? I think you might be missing this commit. The workaround would be to change the file to:
- if: '[ ! -f "/run/elemental/recovery_mode" ]'
name: "Layout configuration"
files:
- path: /etc/elemental/config.d/layout.yaml
.....
elemental:~ # elemental version
v2.1.0+ga1a22e8
Applying the workaround, can't see real difference and still the same pb with /etc/rancher
elemental:~ # mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=385324k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p4 on /run/initramfs/elemental-state type ext4 (rw,relatime)
/run/initramfs/elemental-state/.snapshots/1/snapshot.img on / type ext2 (ro,relatime)
/dev/mmcblk0p2 on /oem type ext4 (rw,relatime)
/dev/mmcblk0p1 on /run/elemental/efi type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p5 on /run/elemental/persistent type ext4 (rw,relatime)
tmpfs on /run/elemental/overlay type tmpfs (rw,relatime,size=481656k,inode64)
overlay on /var type overlay (rw,relatime,lowerdir=/sysroot/var,upperdir=/run/elemental/overlay/var.overlay/upper,workdir=/run/elemental/overlay/var.overlay/work)
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/elemental/overlay/etc.overlay/upper,workdir=/run/elemental/overlay/etc.overlay/work)
overlay on /srv type overlay (rw,relatime,lowerdir=/sysroot/srv,upperdir=/run/elemental/overlay/srv.overlay/upper,workdir=/run/elemental/overlay/srv.overlay/work)
overlay on /home type overlay (rw,relatime,lowerdir=/sysroot/home,upperdir=/run/elemental/persistent/.state/home.overlay/upper,workdir=/run/elemental/persistent/.state/home.overlay/work)
overlay on /opt type overlay (rw,relatime,lowerdir=/sysroot/opt,upperdir=/run/elemental/persistent/.state/opt.overlay/upper,workdir=/run/elemental/persistent/.state/opt.overlay/work)
overlay on /root type overlay (rw,relatime,lowerdir=/sysroot/root,upperdir=/run/elemental/persistent/.state/root.overlay/upper,workdir=/run/elemental/persistent/.state/root.overlay/work)
overlay on /etc/systemd type overlay (rw,relatime,lowerdir=/sysroot/etc/systemd,upperdir=/run/elemental/persistent/.state/etc-systemd.overlay/upper,workdir=/run/elemental/persistent/.state/etc-systemd.overlay/work)
overlay on /etc/ssh type overlay (rw,relatime,lowerdir=/sysroot/etc/ssh,upperdir=/run/elemental/persistent/.state/etc-ssh.overlay/upper,workdir=/run/elemental/persistent/.state/etc-ssh.overlay/work)
overlay on /var/log type overlay (rw,relatime,lowerdir=/sysroot/var/log,upperdir=/run/elemental/persistent/.state/var-log.overlay/upper,workdir=/run/elemental/persistent/.state/var-log.overlay/work)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19469)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=963308k,nr_inodes=1048576,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=192660k,nr_inodes=48165,mode=700,inode64)
elemental:~ # journalctl -u elemental-setup-rootfs
an 08 12:00:24 recovery.lan systemd[1]: Starting Elemental system early rootfs setup...
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Starting elemental version v2.1.0
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Reading configuration from '/etc/elemental'
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Running stage: rootfs.before
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
Jan 08 12:00:24 recovery.lan elemental[658]: WARN[2024-01-08T12:00:24Z] (conditional) Skip 'Skipping stage (if statement error: failed to run cat /proc/cmdline | grep -q "CDLABEL" || cat /proc/cmdline | grep -q "elemental.disable" || cat /proc/cmdline | grep -q "rd.cos.disable"
Jan 08 12:00:24 recovery.lan elemental[658]: : exit status 1)' stage name: Identify live mode
Jan 08 12:00:25 recovery.lan elemental[658]: ERRO[2024-01-08T12:00:25Z] no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[658]: WARN[2024-01-08T12:00:25Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/elemental/live_mode" ]: exit status 1)' stage name: Include cloud init files from the ISO root
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Processing stage step 'Layout configuration'. ( commands: 0, files: 1, ... )
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.before
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
Jan 08 12:00:25 recovery.lan elemental[658]: WARN[2024-01-08T12:00:25Z] 1 error occurred:
Jan 08 12:00:25 recovery.lan elemental[658]: * no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[658]:
Jan 08 12:00:25 recovery.lan systemd[1]: Finished Elemental system early rootfs setup.
At end here is my working conf :
# Layout Elemental OEM configuration file
#
# This file is part of Elemental and will get reset during upgrades.
#
# Before you change this file manually,
# consider copying this file to /usr/local/cloud-config or
# copy the file with a prefix starting by 90, e.g. /oem/91_custom.yaml
name: "Rootfs Layout Settings"
stages:
rootfs:
- if: '[ ! -f "/run/elemental/recovery_mode" ]'
name: "Layout configuration"
files:
- path: /etc/elemental/config.d/layout.yaml
content: |
mount:
ephemeral:
type: tmpfs
size: 25%
initramfs:
- if: '[ ! -f "/run/elemental/recovery_mode" ]'
name: "Rancher Setting"
commands:
- |
mkdir /oem/rancher
ln -s /oem/rancher /etc
I will dust off my raspberry and try to debug this today! You might be hitting a bug with loading the config..
Also the /oem/rancher
symlink was inventive! :+1:
@bobmorane83 I was able to boot my RPi4 and get a persistent path for /etc/rancher
using the following config:
# Rootfs Elemental OEM configuration file
#
# This file is part of Elemental and will get reset during upgrades.
#
# Before you change this file manually,
# consider copying this file to /usr/local/cloud-config or
# copy the file with a prefix starting by 90, e.g. /oem/91_custom.yaml
name: "Rootfs Layout Settings"
stages:
rootfs.before:
- name: "Pull data from provider"
datasource:
providers: ["aws", "gcp", "openstack", "cdrom"]
path: "/oem"
initramfs:
- if: '[ ! -f "/run/elemental/recovery_mode" ]'
name: "Persist /etc/machine-id"
commands:
- |
# persist machine-id
if [ -s /run/elemental/persistent/etc/machine-id ]; then
cat /run/elemental/persistent/etc/machine-id > /etc/machine-id
else
mkdir -p /run/elemental/persistent/etc
cp /etc/machine-id /run/elemental/persistent/etc
fi
rootfs:
- if: '[ ! -f "/run/elemental/recovery_mode" ]'
name: "Layout configuration"
files:
- path: /run/elemental/config.d/layout.yaml
content: |
mount:
write-fstab: true
ephemeral:
type: tmpfs
size: 25%
paths: ['/etc', '/var', '/srv']
persistent:
mode: bind
paths:
- /etc/systemd
- /etc/rancher
- /etc/ssh
- /etc/iscsi
- /etc/cni
- /home
- /opt
- /root
- /usr/libexec
- /usr/local
- /var/log
- /var/lib/elemental
- /var/lib/rancher
- /var/lib/kubelet
- /var/lib/NetworkManager
- /var/lib/cni
- /var/lib/calico
- if: '[ -f "/run/elemental/recovery_mode" ]'
name: "Layout configuration for recovery"
files:
- path: /run/elemental/config.d/layout.yaml
content: |
mount:
write-fstab: true
ephemeral:
type: tmpfs
size: 25%
But used the tumbleweed changes as well.. Let me know if you are still having trouble!
I'll try that. Thanks.
Hi all, (BTW I'am totally newbee to elemental ...)
elemental-toolkit version: ❯ docker run -it --rm ghcr.io/rancher/elemental-toolkit/elemental-cli:latest version v1.3.0+g56b2723
CPU architecture, OS, and Version: On my RPi : Linux pi 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux Same on my Mac M1
Describe the bug ❯ docker run -it --rm ghcr.io/rancher/elemental-toolkit/elemental-cli:latest build-iso --config-dir=./ rpi INFO[2024-05-16T13:49:52Z] Starting elemental version v1.3.0
INFO[2024-05-16T13:49:52Z] Reading configuration from './'
INFO[2024-05-16T13:49:52Z] Preparing squashfs root (1 source)...
INFO[2024-05-16T13:49:52Z] Copying rpi:latest source...
ERRO[2024-05-16T13:50:05Z] Failed installing OS packages: GET https://index.docker.io/v2/library/rpi/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/rpi Type:repository]]
To Reproduce ❯ docker run -it --rm ghcr.io/rancher/elemental-toolkit/elemental-cli:latest build-iso --config-dir=./ rpi
I'am trying to generate a Raspberry Pi 4 image to boot on, but seems that the repositoty is not available anymore.
Did I missed something ? "docker build --build-arg TOOLKIT_REPO=ghcr.io/rancher/elemental-toolkit/elemental-cli --build-arg VERSION=latest -t rpi ." was ok.
Any clue ? Thanks, BR,