OSTree extlinux.conf on Nvidia Jetson Xavier

halverneus commented 3 years ago

My goal is to create an image that can be flashed to an Nvidia Jetson Xavier device that is based on OSTree. I use the script, below, to create the rootfs. According to the Nvidia Documentation the device requires the kernel, devicetree and initrd to be referenced by /boot/extlinux/extlinux.conf. After running the below script, this file does not exist.

My questions are:

[ ] Does OSTree currently support using extlinux.conf as the boot configuration file?
[ ] If not, can I get a pointer into the OSTree code where I might be able to implement this change (perhaps with minor guidance)?
[ ] If so, what am I not doing right or misunderstanding about creating the rootfs?

#!/bin/bash

set -euo pipefail # Exits script on any error.

# To run this script a few base packages are required. Run the following command
# to install this base packages:
#
# apt install -y build-essential debootstrap qemu-user-static ostree
#
# Requires file Jetson_Linux_R32.4.4_aarch64.tbz2 to be in the same directory as
# this setup.sh script.

HOMEDIR="/home/$USER"
echo "Will be using SSH key from the following home directory: $HOMEDIR"

# Start fresh and remove any existing builds. Existing root filesystems may have
# the immutable flag set, so the flag needs to be cleared before the directory
# can be removed.
sudo chattr -i out/Linux_for_Tegra/rootfs/ostree/deploy/mine/deploy/* || /bin/true
sudo rm -rf out || /bin/true
mkdir out
cd out

# Extract the Jetson tools
tar -xf ../Jetson_Linux_R32.4.4_aarch64.tbz2
cd Linux_for_Tegra

# Build the filesystem.
sudo qemu-debootstrap --arch=arm64 bionic rootfs

sudo /bin/bash -c "echo \"deb http://ports.ubuntu.com/ubuntu-ports bionic main universe\" > rootfs/etc/apt/sources.list"

sudo chroot rootfs /bin/bash -c "apt update && apt install -y libdrm-common libdrm2 libegl-mesa0 libegl1 libgbm1 libglapi-mesa libglvnd0 libwayland-client0 libwayland-server0 libx11-xcb1 libxau6 libxcb-dri2-0 libxcb-dri3-0 libxcb-present0 libxcb-sync1 libxcb-xfixes0 libxcb1 libxdmcp6 libxshmfence1 multiarch-support device-tree-compiler fontconfig fontconfig-config fonts-dejavu-core libcairo2 libdatrie1 libevdev2 libfontconfig1 libfreetype6 libgles2 libgraphite2-3 libgudev-1.0-0 libharfbuzz0b libinput-bin libinput10 libjpeg-turbo8 libmtdev1 libpango-1.0-0 libpangoft2-1.0-0 libpixman-1-0 libpng16-16 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libthai-data libthai0 libunwind8 libwacom-bin libwacom-common libwacom2 libwayland-cursor0 libwayland-egl1-mesa libx11-6 libx11-data libxcb-render0 libxcb-shm0 libxext6 libxkbcommon0 libxrender1 python python-minimal python2.7 python2.7-minimal gstreamer1.0-plugins-base iso-codes libasound2-data libcdparanoia0 libogg0 libopus0 liborc-0.4-0 libtheora0 libvisual-0.4-0 libvorbis0a libvorbisenc2 libgstreamer-plugins-bad1.0-0 libasound2 libpangocairo-1.0-0 openssh-server network-manager ostree && rm -rf /var/lib/apt/lists/*"

sudo mkdir -p rootfs/root/.ssh && sudo cp $HOMEDIR/.ssh/id_rsa.pub rootfs/root/.ssh/authorized_keys

sudo ./apply_binaries.sh
sudo rm -f rootfs/lib/systemd/system/nv-oem-config*.*
sudo rm -f rootfs/usr/sbin/nv-oem-config-firstboot
sudo rm -f rootfs/etc/systemd/nv-oem-config*.sh
sudo rm -f rootfs/etc/nv-oem-config.conf.*
sudo rm rootfs/README.txt

# Adapt filesystem to OSTree
# - /bin -> /usr/bin
sudo mv rootfs/bin/* rootfs/usr/bin/ && sudo rmdir rootfs/bin && sudo ln -s /usr/bin rootfs/bin
# - /sbin -> /usr/sbin
sudo mv rootfs/sbin/* rootfs/usr/sbin/ && sudo rmdir rootfs/sbin && sudo ln -s /usr/sbin rootfs/sbin
# - /home -> /var/home
sudo mkdir rootfs/var/home && sudo rmdir rootfs/home && sudo ln -s /var/home rootfs/home
# - /lib -> /usr/lib
sudo mv rootfs/lib/aarch64-linux-gnu/* rootfs/usr/lib/aarch64-linux-gnu/ && sudo rmdir rootfs/lib/aarch64-linux-gnu
sudo mv rootfs/lib/systemd/* rootfs/usr/lib/systemd/ && sudo rmdir rootfs/lib/systemd
sudo mv rootfs/lib/* rootfs/usr/lib/ && sudo rmdir rootfs/lib && sudo ln -s /usr/lib rootfs/lib
# - /opt -> /var/opt
sudo mv rootfs/opt/* rootfs/var/opt/ && sudo rmdir rootfs/opt && sudo ln -s /var/opt rootfs/opt
# - /root -> /var/roothome
sudo mv rootfs/root rootfs/var/roothome && sudo ln -s /var/roothome rootfs/root
# - /media -> /run/media
sudo rmdir rootfs/media && sudo ln -s /run/media rootfs/media
# - /etc -> /usr/etc
sudo mv rootfs/etc rootfs/usr/etc

# Setup OSTree-Systemd tmpfiles.
sudo bash -c 'cat <<EOF > rootfs/usr/lib/tmpfiles.d/ostree.conf
L /var/home - - - - ../sysroot/home
d /var/roothome 0700 root root -
d /var/opt 0755 root root -
d /var/local 0755 root root -
d /run/media 0755 root root -
EOF'

# Move Linux, initrd and devicetree to /usr/lib/modules/${version}
modules="rootfs/usr/lib/modules/4.9.140-tegra"
sudo mv rootfs/boot/tegra194-p2888-0001-p2822-0000.dtb ${modules}/devicetree
sudo mv rootfs/boot/Image ${modules}/vmlinuz
sudo mv rootfs/boot/initrd ${modules}/initramfs.img

# Clear /dev so the OS can be commited.
sudo rm -rf rootfs/dev/*

# Create a repo and commit the OSTree image.
sudo mkdir repo
sudo ostree init --repo=repo --mode=archive
sudo ostree --repo=repo commit --branch=mine rootfs

# Create a fresh rootfs.
sudo mv rootfs orig_rootfs
sudo mkdir rootfs

# Create the OSTree image.
sudo ostree admin init-fs rootfs
sudo ostree --sysroot=rootfs admin os-init mine
sudo ostree --repo=rootfs/ostree/repo remote add mine http://10.0.1.4:8080 --no-gpg-verify
sudo ostree --repo=rootfs/ostree/repo pull-local --remote=mine repo mine
sudo ostree --sysroot=rootfs admin deploy --os=mine mine:mine

# Inform user of next step.
echo "To flash device run: sudo ./flash.sh jetson-agx-xavier-devkit mmcblk0p1"

wmanley commented 3 years ago

I've got a patch that implements extlinux.conf support for the Tegra TK-1 and NX. I'll upload it later today when I get a moment.

halverneus commented 3 years ago

@wmanley , nice! Sounds like something I'm going to need. Any special instructions for using it, once it is available? My day is just getting started and I'd love to test it out. I'm guessing I'll need to compile ostree for my build machine (that makes the image) and for the Xavier (to handle upgrades)? Let me know when you have it up and I'll try it out, immediately. Thank you!

wmanley commented 3 years ago

I've raised #2256 with my bootloader patches.

You'll also need to add:

sudo ln -sf ../loader/syslinux.cfg rootfs/boot/extlinux/extlinux.conf

To your # Create the OSTree image. step - after the deploy. It's worth checking that the kargs are being pulled though correctly into the new /boot/loader/syslinux.cfg file.

halverneus commented 3 years ago

@wmanley , related to this goal, but unrelated to OSTree... when I attempt to flash the Xavier device, flash.sh pushes into the rootfs a new copy of the kernel, as well as many other files. Do you have an existing work-around to maintain the integrity of the rootfs while the device is being flashed? I also seem to have an issue, when I manually set the /boot/extlinux/extlinux.conf boot options that I can't get it to boot with the kernel/initrd/devicetree provided by OSTree... only with the /boot/Image that is inserted during the flash.sh process. Do you have a script to work around this issue?

halverneus commented 3 years ago

@wmanley , I must be missing a step, somewhere. Here is exactly the steps I have taken:

git clone https://github.com/stb-tester/ostree.git
cd ostree
git checkout u-boot-extlinux-compatibility

At this point, I created the following Dockerfile to build OSTree:

FROM ubuntu:20.04 as builder

RUN mkdir -p /root/prj
WORKDIR /root/prj

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
    apt-get install -y build-essential dh-autoreconf bison e2fslibs-dev \
                       libgtk2.0-dev liblzma-dev libgpgme-dev libfuse-dev \
                       libsoup2.4-dev && \
    rm -rf /var/lib/apt/lists/*

COPY . .

RUN mkdir -p out && \
    env NOCONFIGURE=1 ./autogen.sh && \
    ./configure --prefix=/usr && \
    make && \
    make install DESTDIR=/root/prj/out

FROM scratch

COPY --from=builder /root/prj/out/ /

I built and installed with:

DOCKER_BUILDKIT=1 docker build --output=./out/ .
sudo cp -r ./out/usr/* /usr/

After having this version of OSTree installed, I ran the same setup script I placed in the issue (no changes to anything, yet) with the anticipation that I would discover a syslinux.cfg in rootfs/boot/loader/. I did not see this file. Am I missing a step or argument?

wmanley commented 3 years ago

@wmanley , related to this goal, but unrelated to OSTree... when I attempt to flash the Xavier device, flash.sh pushes into the rootfs a new copy of the kernel, as well as many other files. Do you have an existing work-around to maintain the integrity of the rootfs while the device is being flashed?

I've not been using flashing. I copy ostree to a running system and convert it from the inside, so I can't help here.

halverneus commented 3 years ago

@wmanley , am I missing a step (or compiler dependency, flag, etc...) to create the syslinux.cfg? Or is it required that a syslinux.cfg exists before OSTree will take it over?

wmanley commented 3 years ago

@wmanley , am I missing a step (or compiler dependency, flag, etc...) to create the syslinux.cfg? Or is it required that a syslinux.cfg exists before OSTree will take it over?

It's required that extlinux.conf exists and then ostree should take it over. If this doesn't work you could try setting sysroot.bootloader to syslinux in the ostree config.

halverneus commented 3 years ago

I've managed to get a deployment to boot into OSTree. When I have it automated, I'll post the resulting script, here, for anyone it might help (and then close the issue).

halverneus commented 3 years ago

So, I got it working (mostly). Final script is below. There are still some issues:

Requires creating a shortcut from /boot/extlinux/extlinux.conf to /boot/loader/syslinux.cfg after deploying (Nvidia flash overwrites the shortcut with the file).
The syslinux.cfg created by OSTree is not bootable on the Xavier (see script for differences).
OSTree deploy requires "\tKERNEL", which isn't compatible with the Xavier (uses LINUX and spaces). The OSTree-created syslinux.cfg creates the line properly, but deploy doesn't work with "LINUX" instead of "KERNEL".
The 'flash' process installs /boot/Image and /boot/initrd and other useless files that cannot be avoided. Working with Nvidia on that.

The script does require an SSH public key to exist so that you can SSH into the device after flashing. Because of these issues, I'm not sure if the pull request is fully complete, @wmanley (namely, the need for "\tKERNEL" and that the created syslinux.cfg isn't bootable).

#!/bin/bash

set -euo pipefail # Exits script on any error.

# To run this script a few base packages are required. Run the following command
# to install this base packages:
#
# apt install -y build-essential debootstrap qemu-user-static ostree
#
# Requires file Jetson_Linux_R32.4.4_aarch64.tbz2 to be in the same directory as
# this setup.sh script.

HOMEDIR="/home/$USER"
echo "Will be using SSH key from the following home directory: $HOMEDIR"

# Start fresh and remove any existing builds. Existing root filesystems may have
# the immutable flag set, so the flag needs to be cleared before the directory
# can be removed.
sudo chattr -i out/Linux_for_Tegra/rootfs/ostree/deploy/mine/deploy/* || /bin/true
sudo rm -rf out || /bin/true
mkdir out
cd out

# Extract the Jetson tools
tar -xf ../Jetson_Linux_R32.4.4_aarch64.tbz2
cd Linux_for_Tegra

# Build the filesystem.
sudo qemu-debootstrap --arch=arm64 bionic rootfs

sudo /bin/bash -c "echo \"deb http://ports.ubuntu.com/ubuntu-ports bionic main universe\" > rootfs/etc/apt/sources.list"

sudo chroot rootfs /bin/bash -c "apt update && apt install -y libdrm-common libdrm2 libegl-mesa0 libegl1 libgbm1 libglapi-mesa libglvnd0 libwayland-client0 libwayland-server0 libx11-xcb1 libxau6 libxcb-dri2-0 libxcb-dri3-0 libxcb-present0 libxcb-sync1 libxcb-xfixes0 libxcb1 libxdmcp6 libxshmfence1 multiarch-support device-tree-compiler fontconfig fontconfig-config fonts-dejavu-core libcairo2 libdatrie1 libevdev2 libfontconfig1 libfreetype6 libgles2 libgraphite2-3 libgudev-1.0-0 libharfbuzz0b libinput-bin libinput10 libjpeg-turbo8 libmtdev1 libpango-1.0-0 libpangoft2-1.0-0 libpixman-1-0 libpng16-16 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libthai-data libthai0 libunwind8 libwacom-bin libwacom-common libwacom2 libwayland-cursor0 libwayland-egl1-mesa libx11-6 libx11-data libxcb-render0 libxcb-shm0 libxext6 libxkbcommon0 libxrender1 python python-minimal python2.7 python2.7-minimal gstreamer1.0-plugins-base iso-codes libasound2-data libcdparanoia0 libogg0 libopus0 liborc-0.4-0 libtheora0 libvisual-0.4-0 libvorbis0a libvorbisenc2 libgstreamer-plugins-bad1.0-0 libasound2 libpangocairo-1.0-0 openssh-server network-manager ostree && rm -rf /var/lib/apt/lists/*"

sudo mkdir -p rootfs/root/.ssh && sudo cp $HOMEDIR/.ssh/id_rsa.pub rootfs/root/.ssh/authorized_keys

sudo ./apply_binaries.sh
sudo rm -f rootfs/lib/systemd/system/nv-oem-config*.*
sudo rm -f rootfs/usr/sbin/nv-oem-config-firstboot
sudo rm -f rootfs/etc/systemd/nv-oem-config*.sh
sudo rm -f rootfs/etc/nv-oem-config.conf.*
sudo rm rootfs/README.txt

# Adapt filesystem to OSTree
# - /bin -> /usr/bin
sudo mv rootfs/bin/* rootfs/usr/bin/ && sudo rmdir rootfs/bin && sudo ln -s /usr/bin rootfs/bin
# - /sbin -> /usr/sbin
sudo mv rootfs/sbin/* rootfs/usr/sbin/ && sudo rmdir rootfs/sbin && sudo ln -s /usr/sbin rootfs/sbin
# - /home -> /var/home
sudo mkdir rootfs/var/home && sudo rmdir rootfs/home && sudo ln -s /var/home rootfs/home
# - /lib -> /usr/lib
sudo mv rootfs/lib/aarch64-linux-gnu/* rootfs/usr/lib/aarch64-linux-gnu/ && sudo rmdir rootfs/lib/aarch64-linux-gnu
sudo mv rootfs/lib/systemd/* rootfs/usr/lib/systemd/ && sudo rmdir rootfs/lib/systemd
sudo mv rootfs/lib/* rootfs/usr/lib/ && sudo rmdir rootfs/lib && sudo ln -s /usr/lib rootfs/lib
# - /opt -> /var/opt
sudo mv rootfs/opt/* rootfs/var/opt/ && sudo rmdir rootfs/opt && sudo ln -s /var/opt rootfs/opt
# - /root -> /var/roothome
sudo mv rootfs/root rootfs/var/roothome && sudo ln -s /var/roothome rootfs/root
# - /media -> /run/media
sudo rmdir rootfs/media && sudo ln -s /run/media rootfs/media
# - /etc -> /usr/etc
sudo mv rootfs/etc rootfs/usr/etc

# Setup OSTree-Systemd tmpfiles.
sudo bash -c 'cat <<EOF > rootfs/usr/lib/tmpfiles.d/ostree.conf
L /var/home - - - - ../sysroot/home
d /var/roothome 0700 root root -
d /var/opt 0755 root root -
d /var/local 0755 root root -
d /run/media 0755 root root -
EOF'

# Move Linux, initrd and devicetree to /usr/lib/modules/${version}
modules="rootfs/usr/lib/modules/4.9.140-tegra"
sudo mv rootfs/boot/tegra194-p2888-0001-p2822-0000.dtb ${modules}/devicetree
sudo mv rootfs/boot/Image ${modules}/vmlinuz
sudo mv rootfs/boot/initrd ${modules}/initramfs.img

# Clear /dev so the OS can be commited.
sudo rm -rf rootfs/dev/*

# Create a repo and commit the OSTree image.
sudo mkdir repo
sudo ostree init --repo=repo --mode=archive
sudo ostree --repo=repo commit --branch=mine rootfs

# Create a fresh rootfs.
sudo mv rootfs orig_rootfs
sudo mkdir rootfs

# Create the OSTree image.
sudo ostree admin init-fs rootfs
sudo ostree --sysroot=rootfs admin os-init mine
sudo ostree --repo=rootfs/ostree/repo remote add mine http://10.0.1.4:8080 --no-gpg-verify
sudo ostree --repo=rootfs/ostree/repo pull-local --remote=mine repo mine
sudo mkdir rootfs/boot/extlinux
sudo mv orig_rootfs/var rootfs/
sudo mv orig_rootfs/boot/extlinux/extlinux.conf rootfs/boot/extlinux/

# Create links
sudo rm -rf rootfs/bin || /bin/true
sudo rm -rf rootfs/sbin || /bin/true
sudo rm -rf rootfs/home || /bin/true
sudo rm -rf rootfs/lib || /bin/true
sudo rm -rf rootfs/opt || /bin/true
sudo rm -rf rootfs/root || /bin/true
sudo rm -rf rootfs/media || /bin/true
sudo rm -rf rootfs/etc || /bin/true
sudo ln -s /usr/bin rootfs/bin
sudo ln -s /usr/sbin rootfs/sbin
sudo ln -s /var/home rootfs/home
sudo ln -s /usr/lib rootfs/lib
sudo ln -s /var/opt rootfs/opt
sudo ln -s /var/roothome rootfs/root
sudo ln -s /run/media rootfs/media
sudo ln -s /usr/etc rootfs/etc

# Set up expected extlinux.conf.
sudo bash -c '
cat  <<EOF > rootfs/boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
    MENU LABEL primary kernel
    KERNEL /boot/Image
    INITRD /boot/initrd
    APPEND quiet
EOF
'

# Complete OS creation.
sudo ostree --sysroot=rootfs admin deploy --os=mine mine:mine
cd rootfs
usr_dir=`echo ostree/deploy/mine/deploy/*/usr`
cd ..
sudo ln -s "/$usr_dir" rootfs/usr

# Fix extlinux.conf (syslinux.cfg).
orig="rootfs/boot/loader/syslinux.cfg"
tmpfile="extlinux.conf"
cat <<EOF > $tmpfile
TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
    MENU LABEL primary kernel
EOF
cat $orig | grep ostree | grep -v LABEL >> $tmpfile
echo "  APPEND quiet" >> $tmpfile
sudo cp $tmpfile $orig
sudo mv $tmpfile rootfs/boot/extlinux/extlinux.conf

# Inform user of next step.
echo "========================================================================="
echo "To flash device run:"
echo "    cd out/Linux_for_Tegra"
echo "    sudo ./flash.sh jetson-agx-xavier-devkit mmcblk0p1"
echo "To complete install after flashing:"
echo "    rm /boot/extlinux/extlinux.conf"
echo "    ln -s /boot/loader/syslinux.cfg /boot/extlinux/extlinux.conf"
echo "========================================================================="

halverneus commented 3 years ago

I thought I got it working, but when I try to deploy, I get a message about not being booted into an OSTree OS (though ostree admin status gives information about the deployment, which is why I thought everything was working) and that I need to select an OS. Selecting 'mine' as the OS (based on the script, above) and manually updating the /boot/extlinux/extlinux.conf and rebooting does not result in the changes from the changeset being applied (adding a single file to /usr/bin/). I'm clearly missing something. If I drop the symlink for /usr, the OS won't boot. What am I missing or misunderstanding? How did you get yours working from inside the booted OS @wmanley ?

halverneus commented 3 years ago

For what it is worth, I haven't actually been fully successful getting this device to boot into an OSTree-based OS. I thought I had for a time, as 'ostree admin status' returned valid output, but when I tried to deploy or upgrade, it would tell me I wasn't booted into an OSTree system. After running 'ostree admin deploy mine:mine --os=mine' on the live system and rebooting, I get the following message:

error: loading sysroot: Unexpected state: /run/ostree-booted found and in / sysroot, but bootloader entry not found

I do see entries in /boot/loader/entries/ and I match the configuration (manually) in my /boot/extlinux/extlinux.conf file before booting. The only thing I can point to as a possible problem is the following that I get during boot in the system logs:

Jan 28 21:29:05 localhost systemd-gpt-auto-generator[4191]: Failed to dissect: Input/output error
Jan 28 21:29:05 localhost systemd[4183]: /ostree/deploy/mine/deploy/2eec...1001.0/usr/lib/systemd/system-generators/systemd-gpt-auto-generator failed with exit status 1.

I'm getting this with a manual build of OSTree 2020.8.

ostreedev / ostree

OSTree extlinux.conf on Nvidia Jetson Xavier #2255