search

helloSystem / ISO

helloSystem Live and installation ISO
https://github.com/helloSystem/
BSD 3-Clause "New" or "Revised" License
807 stars 58 forks source link

Live mode boot slow, needs 4GB+ RAM #4

Closed probonopd closed 2 years ago

probonopd commented 3 years ago

Currently we need at least 4 GB of RAM for the Live system when the booted sytem in theory only needs <1. This is because everything gets copied into RAM at boot time.

I would like to remove the need for so much RAM, e.g., by loop-mounting system.uzip, and merging it with a tmpfs using unionfs, and then chrooting into that.

FuryBSD had a unionfs setup but it was more involved: https://github.com/furybsd/furybsd-livecd/tree/union-tar-nochroot

I'd like the simple version (loop-mount read-only filesystem, overlay it with tmpfs usion unionfs, chroot into that). If it doesn't work, then collaborate with unionfs kernel devs (who maintains unionfs? @mjguzik, @cemeyer?) to make it work.

probonopd commented 3 years ago

Been experimenting in the 12.2-unionfs branch but it is not working.

https://github.com/helloSystem/ISO/blob/dea799d632438cf2d065f03a5a8b62ac595a1359/overlays/ramdisk/init.sh#L33

fails.

Any help appreciated.

probonopd commented 3 years ago

Check what NomadBSD does, as it uses unionfs and hence boots the Live system quickly. (Update: It doesn't seem to run from an ISO, hence it is unclear if it can work as a Live DVD.)

probonopd commented 3 years ago

My experiements in the pure-zfs-test branch resulted in a kernel panic so far: https://github.com/helloSystem/ISO/releases/tag/13.0-pure-zfs-test

Tunas1337 commented 3 years ago

This seems like an issue that, if possible, really ought to be referenced in a little "echo" message in the bootloader, before the system attempts to load further. It's a source of confusion for a lot of people, and I'm sure there are people who have tested helloSystem, seen this, thought it was fundamentally broken and given up on it (myself included, had I not checked the GitHub issues!) EDIT: alternatively, this issue could be pinned for easier discovery, as opposed to searching closed issues to find a reference to this. It'd also stop people from making issues related to it, thus saving time.

probonopd commented 3 years ago

Pinned the issue @Tunas1337

probonopd commented 3 years ago

Thanks to the help of @pkgdemon we have an experimental Live build that should address this issue:

https://github.com/helloSystem/ISO/releases/tag/continuous-12.2-unionfs

Note that this build contains fluxbox rather than helloSystem to make the ISO smaller (and because the 12.2-unionfs branch has diverged a lot from the current experimental branch of helloSystem), but it should be possible to port the changes over to the helloSystem builds in case everything turns out to work well.

So let's test, especially

Username: liveuser, no password

@pkgdemon noted on IRC:

There may be other catches for unionfs such as devmatch not supporting kldload properly that I ran into. One thing that should be testing is to make sure autoloading of ums, if_iwm, if_iwn work with devmatch. In the past with ghostbsd it could also be observed that mate would lock up. I never ran into the lock up issues myself with FuryBSD. I think it just to be evaluated how well it works with Hello. (...) With unionfs you cannot reoot into the memory file system. You chroot. So what might end up happening is that certain low level utilities like kldload if_iwn might not be able to load a firmware like iwm8000Cfw because trying to use the real upper layer kldload /sysroot/boot/kernel/iwm8000Cfw.ko but because kldload is chrooted it needs to do kldload /boot/kernel/iwm8000Cfw.ko is what i ran into. (...) Hopefully those low level issues have been resolved since I've last tried and maybe it would be a good solution for hellosystem.

probonopd commented 3 years ago

I was privately contacted by a helloSystem user who wishes to remain unnamed, and has summarized the issues as follows:

  1. Splash screen doesn't always appear.
    1. The whole system image needs to be decompressed before the user can start using the system. CPU utilization when decompressing system.uzip is pretty high, which can turn it into a bottleneck on computers with slower CPU's and/or can cause the CPU fan to spin at high RPM thus generating unpleasant noise.
    2. The whole system image needs to be read into memory before the user can start using the system which takes time and depends on I/O speed.
    3. The whole system image is kept in memory, which raises the minimum required RAM to 4 GB and reduces the amount of RAM available for normal operation of the system.

The user presented two solutions to problem 2 and noted that 3 and 4 need to be addessed separately and noted:

Both solutions drop uzip, which apart from solving problem 2 above also saves us from kernel panics and hangs caused by or related to uzip (see issue #102). It also makes building helloSystem ISO's quicker. For problems 3 and 4 above using unionfs is not the solution. See the BUGS section in mount_unionfs(8) if you wonder why.

Solution 1 Use as system.img the byte stream generated by 'zfs send'. It contains only the data from the source zpool, without any free disk space. When booting, you create a 3 GB memory disk, then create a zpool on that memory disk and then you use 'zfs recv' to fill in the data. This solution is already in use by GhostBSD so you can take a look at their scripts. The increase in size measured on one recent helloSystem .iso is 10.5% or 174 MB. With Zstandard compression you will likely get even better results.

Solution 2 Have system.img be a disk image of the sole disk of a zpool which is 1.6 GB in size and is full of 1.6 GB of useful data. On boot, copy system.img to a memory disk (using simple dd; there is no need to treat it as a device and to use zfs send | zfs recv) to make it read-write. Then add a second memory disk to the pool which will contain all the free disk space in the pool. Care may need to be taken to avoid ZFS seeing two zpools with the same GUID's.

For problem 4 the use of swap space was suggested, which unfortunately does not work on truly read-only media such as DVD.

So the solutions suggested by the unnamed user appear to be (partial) alternatives to what @pkgdemon has implemented and what seems to address at least 3 and 4.

Probably all solutions come with specific pros and cons which I currently do not fully understand yet. So some experimentation will be needed to see what works best for helloSystem.

Thank you everyone sharing your ideas and code! :+1:

probonopd commented 3 years ago

Porting @pkgdemon's commits to the experimental-unionfs branch that is up-to-date to the experimental branch I noticed the following:

If we can get everything else to behave properly, those trade-offs are probably worth it.

Further results with the experimental-unionfs branch ISOs:

probonopd commented 3 years ago

On Intel GPU machines I cannot boot helloSystem ISOs from the experimental-unionfs branch all the way to the desktop. Things get stalled shortly after Starting devd and Autoloading module: if_iwm.ko. https://github.com/helloSystem/ISO/releases/tag/experimental-unionfs-12.2

However, the non-helloSystem build from the 12.2-unionfs branch https://github.com/helloSystem/ISO/releases/tag/continuous-12.2-unionfs does boot on those machines, albeit without the proper GPU drivers because initgfx is not on that image.

Possibly we are running into a conflict with initgfx? (Need to find a way to disable it for testing) Possibly it is something else.

@pkgdemon wrote in https://github.com/furybsd/furybsd-livecd/commit/29fb0a18db7e1d44613954b2b9561c0eb1575aec:

  • Also fixes low level utilities such as umount and kldload which do not operate well in chroot.

  • As a result this fixes devmatch.

  • Move early xorg config to reroot init

Maybe we are missing something.

Do you have any ideas @pkgdemon?

probonopd commented 3 years ago

This is what happens with https://github.com/helloSystem/ISO/releases/tag/experimental-unionfs-12.1 (commit 8544ac2) when booted with unset boot_mute on two machines with Intel GPUs, even tough we have initgfx_enable=NO.

Boot stalls just above the section with the red arrows. The section with the red arrows is output of repeatedly pressing Ctrl-T.

image

On this machine (Acer Revo RL85) I get cmd: chmod 911 [ufs]. 911 - does that even exist? On another machine (Acer Travelmate B) I get cmd: chmod 759 [ufs]

The very same ISO boots on a machine with Nvidia GPU (but not using the Nvidia driver). It may or may not be related to GPUs at all. More testing on different systems would be helpful.

crees commented 3 years ago

Just a thought, but we can probably get around the chroot/module issues by nullfs mounting /boot with all of the modules etc onto the chroot's /sysroot/boot, or wherever necessary. Obviously this needs doing before the chroot call.

probonopd commented 3 years ago

Worth a try @pkgdemon?

pkgdemon commented 3 years ago

Interesting. Normally devmatch starts after devd. Wonder what it is trying to chmod? Perhaps booting with verbose mode enabled as well would reveal more? Maybe try boot -v as well @probonopd.

probonopd commented 3 years ago

How did you deduce that it is devmatch that is trying to chmod?

pkgdemon commented 3 years ago

How did you deduce that it is devmatch that is trying to chmod?

I am not sure what is trying to chmod that is that is where the extra verbosity from verbose boot might help reveal what it is I am hoping. I normally would see devmatch do "Autoloading module" multiple times for all the modules it finds directly after devd on a working system, and I am not seeing that here.

probonopd commented 3 years ago

I am not sure what is trying to chmod that is that is where the extra verbosity from verbose boot might help reveal what it is I am hoping. I normally would see devmatch do "Autoloading module" multiple times for all the modules it finds directly after devd on a working system, and I am not seeing that here.

Even booting in verbose mode did not really give me much clue, and Ctrl-T brings up different commands it is stalling on, not always chmod.

Additional observations:

Could initgfx be the culprit?

Unlikely, because it is disabled.

Could the wireless driver firmware loading issue be the culprit?

Unlikely, because the same issue happens on a device with Intel GPU but Broadcom WLAN chip.

(By the way, setting set hint.if_iwm.0.disabled=1 in the bootloader does not stop Autoloading module: if_iwm.ko from happening.)

I am puzzled as to why fluxbox-12.2-RELEASE-96ed308-amd64.iso succeeds but hello-0.5.0_8544ac2e98c21d3c398f33f7829e62cc60cfacc7-FreeBSD-12.1-amd64.iso does not.

Could the Intel GPU driver itself be the culprit?

When fluxbox-12.2-RELEASE-96ed308-amd64.iso runs on Intel GPU machines, the Intel GPU kernel modules (drm.ko, i915kms.ko) are not loaded by default, but using furybsd-xorg-tool to load them works and still results in a working system.

What I will try next:

Port the changes from experimental-unionfs to a new branch of experimental which I will call experimental-unionfs-2, and build 12.2 based images from that branch for both the fluxbox and hello desktops. This way we will have the latest initgfx and Intel drivers on board.

https://cirrus-ci.com/github/helloSystem/ISO/experimental-unionfs-2

probonopd commented 3 years ago

hello-0.5.0_087dc31b09e3627e2e72c439e7c46675b6cbae9e-FreeBSD-12.2-amd64.iso

probonopd commented 3 years ago

fluxbox-cbef0fc5fa7f140885c219058ead745cb0ceaa6f-12.2-amd64.iso.md5 which uses the fluxbox rather than hello configuration does boot on Intel GPU devices and does load the Intel driver using initgfx, although on one device (Acer TravelMate B) the screen is slightly garbled (possibly not really reproducible)... at this point I am really puzzled. Something in the hello vs. fluxbox configuration must be causing it to break, but what?

probonopd commented 3 years ago

With unionfs you cannot reoot into the memory file system. You chroot. So what might end up happening is that certain low level utilities like kldload if_iwn might not be able to load a firmware like iwm8000Cfw because trying to use the real upper layer kldload /sysroot/boot/kernel/iwm8000Cfw.ko but because kldload is chrooted it needs to do kldload /boot/kernel/iwm8000Cfw.ko is what i ran into.

Just a thought, but we can probably get around the chroot/module issues by nullfs mounting /boot with all of the modules etc onto the chroot's /sysroot/boot, or wherever necessary. Obviously this needs doing before the chroot call.

As in

mkdir -p /sysroot/sysroot/boot
mount -t nullfs /boot /sysroot/sysroot/boot

prior to

chroot /sysroot /usr/local/bin/furybsd-init-helper

?

Notice the /sysroot/sysroot. So that after chrooting, we still have /sysroot/boot?

Edit: Yes, but it appears empty.

Need to use

mount -t nullfs /sysroot/boot /sysroot/sysroot/boot

?

probonopd commented 3 years ago

Got it to work - https://github.com/helloSystem/ISO/releases/download/working-experimental-unionfs-2-12.2/hello-0.5.0_0d10343e6301ba30721e4371cc088d460c8db1ad-FreeBSD-12.2-amd64.iso seems to be working :+1:

Will do more testing.

Edit: Working on one of my Intel-based devices... but not on the others... wth

probonopd commented 3 years ago

Well, actually,...

Tried to boot hello-0.5.0_0d10343e6301ba30721e4371cc088d460c8db1ad-FreeBSD-12.2-amd64.iso a couple of times on the same machine, Acer TravelMate B. I counted how often the boot resulted in which situation.

Sometimes the boot works - 2 out of 10 times

On this machine it seems we get stuck somewhere around process ID 79x, although the exact process ID and process name that gets stuck differs from boot to boot (or is this an artefact of Ctrl-T reporting different processes randomly?).

Most often we see:

Holding Ctrl-T pressed down until we get stuck can give us an indication where we are in the boot process. Some of the last processes that are running before we are stuck on chmod include kenv, devd, env, awk, devmatch, asmctl, and devd again. By then pressing ScrLk, we can use PgUp and PgDn to scroll around. Maybe this can help us to find out what is running this chmod command and where exactly we are stuck in the boot process. (Remember the Fluxbox desktop ISO boots fine!)

Sometimes, Processes get stuck in "ufs" state under persistent CPU load.

Here are more boots of the exact same ISO on the exact same machine:

Somehow I suspect the issue has to do with the combination of chroot/unionfs and devmatch. Searching the net, I found at https://github.com/furybsd/furybsd-livecd/releases/tag/12.0-XFCE-12-02-2019-01:

Now using dump restore to clone freebsd to md image, and unionfs for /usr filesystem. This was changed so we can perform a reroot which fixes devmatch using kldload and other scripts using umount which may not operate properly in a chroot.

Maybe @pkgdemon has an idea what might be going wong here?

crees commented 3 years ago

There is also this, which could be related- something about unionfs needing an ls.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=175449

I think the dump/restore to a small / and separate /usr is the way to go.

probonopd commented 3 years ago

@crees, it'd be tremendous if you could find the time at some point to try your hypothesis menioned on IRC:

the main difference appears to be that furybsd does the copy of just / (so that's /etc, /var, /boot and others except /usr), then has the uzip only as /usr / can be really tiny, so only take a few seconds to copy over to a 100MB md Then just mount /usr from the uzip

mdconfig -t swap -s 100M newfs /dev/md0 && mount /dev/md0 /sysroot mount $uzip_image /sysroot/usr mount -o union $persistent_image /sysroot/usr You can then have a symlink of /var to /usr/var Oh, and (cd /sysroot && restore rf /root.dump)

grahamperrin commented 3 years ago

Recalling https://github.com/helloSystem/ISO/issues/55#issuecomment-751874784:

… user modifications in a read-write unionfs overlay …

Is this https://github.com/helloSystem/ISO/issues/4 also partly to provide persistent read-write storage for end users of the live (non-installed) system?

grahamperrin commented 3 years ago

https://github.com/helloSystem/ISO/issues/4#issue-733995681

Currently we need at least 4 GB of RAM

If I'm not mistaken, live boot times increase dramatically when there's more.

probonopd commented 3 years ago

Uh, that would be one more reason to get to another way of booting the Live system. Many machines we are targeting (e.g., older Macs) barely meet the 4GB requirement and may not be easy to upgrade.

probonopd commented 3 years ago

Maybe we have been thinking way too complicated all along.

So, another approach: Setting Up FreeBSD for Read-Only Booting makes it seem like it is possible to boot FreeBSD from read-only media. Looks rather simple!

You want to disable any programs that have to write to the filesystem and provide a memory-backed filesystem for the one area that does need to be written (/var/run). You could also make all of /var an md(4) filesystem but this is not necessary unless you want to run syslogd(8), sendmail(8), etc.

The article suggests that FreeBSD can be booted if /var/run the the only place that is writable.

I suspect that this is not entirely correct, at least not for a modern desktop system with Xorg:

We could probably work around these by symlinking these directories into /var/run/.

Then there is initgfx, which wants to write to, e.g., /usr/local/etc/X11/xorg.conf.d/. So the question is, can we trick Xorg into also using e.g., /var/run/X11/xorg.conf.d/?

New plan of attack: Install helloSystem to hard disk, then find a way to make it work with root_rw_mount="NO" while still having Xorg initgfx working. Once we have accomplished that, let the Live ISO boot into a read-only system, and only once the system is fully booted start adding union mounts if so desired by the user.

Main issue: initgfx seems to require a writable /etc and /usr/local/etc since it is doing things like cp -a "${path_tmp_rc_conf}" /etc/rc.conf, rm -f "/usr/local/etc/libmap.d/nvidia.conf", mkdir /usr/local/etc/libmap.d. So the main issue with getting a read-only system running may actually be initgfx?

pkgdemon commented 3 years ago

Maybe we have been thinking way too complicated all along.

So, another approach: Setting Up FreeBSD for Read-Only Booting makes it seem like it is possible to boot FreeBSD from read-only media. Looks rather simple!

You want to disable any programs that have to write to the filesystem and provide a memory-backed filesystem for the one area that does need to be written (/var/run). You could also make all of /var an md(4) filesystem but this is not necessary unless you want to run syslogd(8), sendmail(8), etc.

The article suggests that FreeBSD can be booted if /var/run the the only place that is writable.

I suspect that this is not entirely correct, at least not for a modern desktop system with Xorg:

  • /var/log also regularly gets written to
  • Xorg seems to need to write to $HOME, e.g., to ~/.Xauthority

We could probably work around these by symlinking these directories into /var/run/.

Then there is initgfx, which wants to write to, e.g., /usr/local/etc/X11/xorg.conf.d/. So the question is, can we trick Xorg into also using e.g., /var/run/X11/xorg.conf.d/?

  • We could symlink /usr/local/etc/X11/xorg.conf.d/ to /var/run/X11/xorg.conf.d/
  • Put the non-initgfx-related files into /usr/local/share/X11/xorg.conf.d/ instead of /usr/local/etc/X11/xorg.conf.d/

New plan of attack: Install helloSystem to hard disk, then find a way to make it work with root_rw_mount="NO" while still having Xorg initgfx working. Once we have accomplished that, let the Live ISO boot into a read-only system, and only once the system is fully booted start adding union mounts if so desired by the user.

Main issue: initgfx seems to require a writable /etc and /usr/local/etc since it is doing things like cp -a "${path_tmp_rc_conf}" /etc/rc.conf, rm -f "/usr/local/etc/libmap.d/nvidia.conf", mkdir /usr/local/etc/libmap.d. So the main issue with getting a read-only system running may actually be initgfx?

@probonopd This is basically what ghostbsd did for years. Except that it would tar specific directories at boot like /etc that had to be writable, mount this as tmpfs, and untar the small archive over the top. The rc.initdiskless script if used will already make /var and /tmp writable. If you need additional directories tar is probably the way to go. I don't know why you would want rw_root_mount="NO" after install though? Yes the main issue with a read only system would be things like initgfx unless you can punch through whatever directories it needs. The next issue would be not allowing the user to instal packages or configure other parts of the system for example if you didn't ship a driver they needed. Depending on your desired goals I think readonly sounds like a great solution for you.

kettle-7 commented 3 years ago

Would it be possible to make a ufs filesystem in a file on the USB, and mount that read-write? It would make the ISO bigger but you'd have less RAM taken up.

probonopd commented 3 years ago

Well. How do you mount a ufs file that resides on a read-only medium such as an ISO read-write? That is the 100 points question. I am trying something at the moment but it is not ready yet.

kettle-7 commented 3 years ago

I don't want to seem like I'm pulling Linux into this, but isn't that how Ubuntu and friends do it? I'll have a better look, but it seems like they get the read-only squashfs filesystem and make it writable.

Or you could mount the root readonly, and then mount tmpfs on /var/tmp, /var/run, /var/log, /etc (and on FreeBSD you'd need /usr/etc and /usr/local/etc, but on Ubuntu those are subdirectories of /etc), and /home.
In order to make that simpler, you could just mount tmpfs on /var and do the Fedora Silverblue thing of making all of the writable directories (including /home) subdirectories of /var.

probonopd commented 3 years ago

I am excited to share the news that a new FreeBSD kernel module is in the works that will allow us to do what we need, combine a read-only with a read-write disk to get a read-write disk: geom_rowr by Jordan Gordeev. I am preparing an ISO for testing that will use it for booting in the geom_rowr branch.

Edit: We now have a working ISO for the geom_rowr tree. Known issue: Cannot be installed yet because the installer has not yet been reworked.

This is the diff: https://github.com/helloSystem/ISO/compare/helloSystem:6139ff2...3aa68d9?expand=1

probonopd commented 3 years ago

Please do test https://github.com/helloSystem/ISO/releases/tag/geom_rowr-12.2 and let me know whether it boots on your machine. I would be especially interested in 2-3 GB RAM machines. (Do not attempt to install it.)

Tunas1337 commented 3 years ago

let me know whether it boots on your machine. My results are as follows:

  • [x] Attempted in a VirtualBox VM with 2 GB RAM - boots
  • [x] Attempted in a VirtualBox VM with 3 GB RAM - boots
  • [x] Attempted in a QEMU VM with 2 GB RAM - boots

Screenshot with htop output on a 2GB VBox VM (which also shows a bug where the dock does not listen to resolution changes - has that been reported yet?): image

I'd say it's a resounding success.

probonopd commented 3 years ago

Thanks @Tunas1337. I think the remaining issues with VMs are not related to geom_rowr and should be addressed separately. We are mostly interested in supporting real ("bare metal") hardware though.

grahamperrin commented 3 years ago

https://github.com/helloSystem/ISO/issues/4#issuecomment-832918927

dock does not listen to resolution changes

https://github.com/helloSystem/Dock/issues/5

(Not an issue with VirtualBox.)

Tunas1337 commented 3 years ago

We are mostly interested in supporting real ("bare metal") hardware though.

I'm aware; sadly I have no machines with <4GB RAM right now. PS: Somehow, in my previous comment I neglected to mention the most important thing, that the implementation of geom_rowr reduced boot times to 30-45 seconds from several minutes as seen before. That was the thing I wanted to report in the first place, and that was what I was testing with the VMs. D'oh!

kettle-7 commented 3 years ago

geom_rowr booted great, although because of an issue with my USB stick (I need to go down to the shop and get one that works properly) after about 10 minutes I got I/O errors. All. Of. The. Time.

But that's most likely only a problem that affects me, so it's great.

probonopd commented 3 years ago

after about 10 minutes I got I/O errors

Please try with another USB stick. It's surprising how many go bad relatively quickly, even brand ones...

probonopd commented 3 years ago

Note to self: Directly after booting, one can use find . -cmin -10 to get a list of files that were written to in the last 10 minutes (= since booting).

It looks like these things need to be writeable in order to boot into Xorg with graphics autodetection using initgfx (... = multiple files in this directory):

/
/var
/var/db
/var/db/entropy
/var/db/entropy/saved-entropy.1
/var/db/pkg
/var/db/pkg/local.sqlite
/var/db/fontconfig
/var/db/fontconfig/...
/var/db/zoneinfo
/var/log/...
/var/spool/lock
/var/spool/lock/clean_var
/var/msgs
/var/msgs/bounds
/var/tmp
/var/run
/var/run/dhclient
/var/run/dhclient/dhclient.wlan0.pid
/var/run/wpa_supplicant
/var/run/wpa_supplicant/wlan0
/var/run/wpa_supplicant/wlan0.pid
/var/run/clean_var
/var/run/ld-elf32.so.hints
/var/run/dbus
/var/run/dbus/system_bus_socket
/var/run/dbus/pid
/var/run/cups
/var/run/cups/certs
/var/run/cups/certs/0
/var/run/cups/cupsd.pid
/var/run/cups/cups.sock
/var/run/ConsoleKit
/var/run/ConsoleKit/database
/var/run/ld-elf.so.hints
/var/run/devd.pipe
/var/run/devd.seqpacket.pipe
/var/run/devd.pid
/var/run/webcamd.0.4.0.pid
/var/run/automount.state
/var/run/dsbdriverd.pid
/var/run/syslogd.sockets
/var/run/log
/var/run/logpriv
/var/run/syslog.pid
/var/run/automountd.pid
/var/run/autounmountd.pid
/var/run/dmesg.boot
/var/run/os-release
/var/run/utx.active
/var/run/avahi-daemon
/var/run/avahi-daemon/pid
/var/run/avahi-daemon/socket
/var/run/slim.pid
/var/run/cron.pid
/var/run/consolekit.pid
/var/run/user
/var/run/user/1000
/var/run/slim.auth
/var/run/moused.ums0.pid
/var/lib
/var/lib/xkb
/var/lib/dbus
/var/lib/dbus/machine-id
/var/lib/asmctl.conf
/var/initgfx_config.id
/boot
/boot/modules
/boot/modules/linker.hints
/boot/entropy
/tmp/...
/root
/root/.cache
/root/.cache/mesa_shader_cache
/root/.cache/mesa_shader_cache/index
/root/.cache/mesa_shader_cache/...
/usr/local/include
/usr/local/libdata/pkgconfig
/usr/local/share
/usr/local/share/doc
/usr/local/share/licenses
/usr/local/share/X11/xorg.conf.d
/usr/local/bin
/usr/local/lib
/usr/local/lib/xorg/modules/drivers
/usr/local/lib/xorg/modules/input
/usr/local/man/man1
/usr/local/man/man4
/usr/local/etc
/usr/local/etc/xdg/autostart
/usr/local/etc/rc.d
/usr/local/etc/pam.d
/usr/local/etc/X11/xorg.conf.d
/usr/local/etc/X11/xorg.conf.d/10-video-initgfx.conf
/usr/local/etc/printcap
/usr/local/sbin
/usr/local/var
/usr/local/var/localize
/usr/local/var/localize/include
/usr/home/liveuser/...
/sbin/mount
/sbin/mount_udf
/sbin/mount_cd9660
/sbin/mount_nullfs
/sbin/mdmfs
/sbin/mount_unionfs
/sbin/mount_mfs
/sbin/mount_fusefs
/sbin/mount_nfs
/sbin/mount_msdosfs
/etc
/etc/motd
/etc/X11
/etc/X11/xorg.conf
/etc/rc.conf
/etc/hostid
/etc/host.conf
/etc/localtime
/net
/entropy
probonopd commented 3 years ago

Issues with geom_rowr hello-0.5.0_acc54acd1e3ae51f6f2eaa177dbba47e9cd261c4-FreeBSD-12.2-amd64.iso:

To be verified wether these things work correctly with the equivalent non-geom_rowr build 0E212.

probonopd commented 3 years ago

Tests show that geom_rowr is useful but may possibly be overkill for just getting the ISO booted into a graphical desktop and comes with its own caveats (such as the ones in the post directly above).

Need to check out how we can use /etc/rc.initdiskless to get enough of FreeBSD read-write so that we can boot into a graphical desktop without needing any extra kernel modules. Possibly we can use it to make the files/directories listed above r/w?

The book Absolute FreeBSD by Michael Lucas, describes it like this:

The diskless system checks the file /conf/base/etc/diskless_remount for a list of directories it should mount as memory filesystems. Without this file, no memory filesystems get created, and your diskless host shares a single read-only userland with all of the other diskless hosts. The diskless_remount file contains a list of filesystems to be remounted.

So /etc/rc.initdiskless was designed with systems booting from a NFS network server but we want to use it for booting from a local r/o medium and the question is, can we use /etc/rc.initdiskless for this purpose.

Does this mean that we can just create the file /conf/base/etc/diskless_remount with the content /etc, run /etc/rc.initdiskless, and expect to get a read-write version of /etc?

Apparently no:

FreeBSD% sudo sh /etc/rc.initdiskless 
mount_nfs: no <host>:<dirpath> nfs-name
mount_nfs /etc /conf/base/etc failed: dropping into /bin/sh

Looks like it is hardwired to use NFS whereas we do not want to use NFS because we are using a local rather than a network filesystem.

grahamperrin commented 3 years ago

https://github.com/helloSystem/ISO/issues/4#issuecomment-842745565

… issue with my USB stick …

FWIW I habitually use StressDisk to test disks before using them for anything of value.

sysutils/stressdisk | https://lists.freebsd.org/pipermail/freebsd-questions/2021-August/294600.html

HDAT2 excels, but it's not available for FreeBSD.

probonopd commented 3 years ago

Looks like @mszoek has got unionfs based Live ISOs to work properly for Airyx. It boots way faster and needs less RAM than helloSystem does at the moment.

Are there downsides, such as no more automatic kernel module loading? (Still to be checked.)

mszoek commented 3 years ago

Looks like @mszoek has got unionfs based Live ISOs to work properly for Airyx. It boots way faster and needs less RAM than helloSystem does at the moment.

"Properly" remains to be seen :)

Are there downsides, such as no more automatic kernel module loading? (Still to be checked.)

I believe so from what we've seen so far. It also ran out of disk (overlay FS) running from the LiveCD overnight.

ghost commented 3 years ago

Are there any test builds of helloSystem v0.6.0 for RAM < 2 GB?

probonopd commented 3 years ago

No.

probonopd commented 2 years ago

Presenting:

The FreeBSD read-only Xorg challenge

We need to find a way to boot into a fully working Xorg desktop (including working initgfx) on a read-only system. This would greatly improve our Live ISO options, and would also be the foundation for a future image-based system where the operating system itself could stay inside a read-only image file.

My earlier experiments at achieving this resulted in a lot of failures and frustration, with countless half-working or not-working Live ISOs build.

So there is another approach about trying this.

You will notice that the system can no longer boot into a working Xorg desktop.

Now our task is to find a way of strategically creating unionfs, nullfs, tmpfs in the right places and/or copying/moving files around until we find a way to get into a graphical desktop again.

The following leads to an instant reboot:

sudo mount -t unionfs /tmp /var/log
# sudo mount -t unionfs /tmp /var/lib
# sudo mount -t unionfs /tmp /etc
startx

Why?

A working solution

The following seems to work:__

Change the localize_start() function in /usr/local/etc/rc.d/localize like this:

localize_start()
{

mount -t tmpfs tmpfs /tmp # We need more space than what is the default
mount -t tmpfs tmpfs /var/log

cp -R /usr/local/etc/X11/xorg.conf.d /tmp
mount -t tmpfs tmpfs /usr/local/etc/X11/xorg.conf.d
mv /tmp/xorg.conf.d/* /usr/local/etc/X11/

mount -t tmpfs tmpfs /usr/local/var # FIXME: localize
mkdir -p /usr/local/var/db /usr/local/var/lib

# FIXME: Do more elegantly; e.g., by creating a new home on the fly from skel
cp -R /usr/home /tmp
mount -t tmpfs tmpfs /usr/home
mv /tmp/home/* /usr/home/
rm -f /usr/home/user/.Xauthority*
chown -R user /home/user

/usr/local/sbin/localize

}

(Of course this should eventually go into its own rc file but putting it into the existing one was the quickest way for me to ensure it runs at the correct place within the rc order.)

With this in place, the mounts on my test system are looking like this (notice those nice read-only):

zroot/ROOT/default on / (zfs, local, noatime, read-only, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, read-only, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, read-only, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, read-only, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, read-only, nfsv4acls)
zroot on /zroot (zfs, local, noatime, read-only, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, read-only, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, read-only, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, read-only, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, read-only, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, read-only, nfsv4acls)
tmpfs on /var (tmpfs, local)
map -hosts on /net (autofs)
tmpfs on /tmp (tmpfs, local)
tmpfs on /tmp (tmpfs, local)
tmpfs on /var/log (tmpfs, local)
tmpfs on /usr/local/etc/X11/xorg.conf.d (tmpfs, local)
tmpfs on /usr/local/var/db (tmpfs, local)
tmpfs on /usr/home (tmpfs, local)

Now that we have a baseline to work from we should be able to experiment more, e.g., also with selectively using unionfs.

probonopd commented 2 years ago

Actually, now we have proof that it is possible tor run a graphical desktop from a fully read-only medium, we can rethink the whole Live ISO architecture.

Currently we use:

Can we do without most of this and instead boot directly into a uzip? Should be doable according to its manpage:

 geom_uzip is not limited to supporting only md(4) images.  The image can
 also reside on a block device.  (For example, a disk, USB flash drive,
 DVD-ROM, etc).  The appropriate device node will appear with the .uzip
 suffix.

 geom_uzip allows mounting the root file system from a compressed disk
 partition by setting the vfs.root.mountfrom tunable.  See loader.conf(5)
 for details.

Sounds exactly like what we need, doesn't it? Will try this path now.

probonopd commented 2 years ago

Current line of thinking is to have an rc script very early in the boot process inside the compressed filesystem image that kicks off the "make writable the essential locations" process, so that no scripts/initial ramdisks outside of the compressed filesystem are needed to do trigger this. This allows us to just tell the kernel to boot from the compressed filesystem, and once that is booting, it will make parts of itself r/w "by itself".

Like this /etc/rc.d/live:

#!/bin/sh

# PROVIDE: live
# REQUIRE: sysctl
# BEFORE: hostid

. /etc/rc.subr

name="live"
desc="Prepare read-only filesystem for live boot"
stop_cmd=":"
start_cmd="live_start"

live_start()
{
    mount -t devfs devfs /dev
    mount -t tmpfs tmpfs /tmp
    mount -t tmpfs tmpfs /media
    mount -t tmpfs tmpfs /var/log
    cp -R /usr/local/etc/X11/xorg.conf.d /tmp
    mount -t tmpfs tmpfs /usr/local/etc/X11/xorg.conf.d # initgfx writes there
    mv /tmp/xorg.conf.d/* /usr/local/etc/X11/
    mount -t tmpfs tmpfs /usr/local/var # FIXME: localize
    mkdir -p /usr/local/var/db /usr/local/var/lib
    # FIXME: Do more elegantly; e.g., by creating a new home on the fly from skel
    cp -R /usr/home /tmp
    mount -t tmpfs tmpfs /usr/home
    mv /tmp/home/* /usr/home/
    rm -f /usr/home/liveuser/.Xauthority* 2>/dev/null || true
    chown -R liveuser /home/liveuser
    # TODO: Port furybsd-init-helper into preinit.sh
    /usr/local/bin/furybsd-init-helper 
}

load_rc_config $name
run_rc_command "$1"

Also, the compressed filesystem image must not contain unsuitable entries in /etc/fstab, so we clear it out.

/boot and the compressed filesystem image do not need to be on the same physical medium. For example, I am using one USB stick tor the bootable filesystem that holds /boot and another USB stick that holds the parition for the compressed filesystem image. (You do not need to do it this way. I just found it more convenient for development.)

By the way, a fast way to make changes on the compressed filesystem image is:

# Make changes to the compressed filesystem image by using unionfs on the dev machine
mount -o ro -t ufs  /dev/da1p1.uzip /mnt
mkdir -p /tmp/union
mount -t unionfs /tmp/union/ /mnt/etc/
echo "" > /mnt/etc/fstab
nano /mnt/etc/rc.d/live
chmod +x /mnt/etc/rc.d/live

# Create new compressed filesystem image
makefs -o label=live /home/user/Desktop/system.ufs /mnt  # label is later passed to the kernel for booting
mkuzip -o /home/user/Desktop/system.uzip /home/user/Desktop/system.ufs
umount /mnt/etc/
umount /mnt/

# Write new compressed filesystem image to dedicated disk (allowing for a quick development cycle)
umount /dev/da1*
/sbin/gpart destroy -F da1
/sbin/gpart create -s GPT da1
BYTES=$(ls -la /home/user/Desktop/system.uzip | awk '{print $5}')
/sbin/gpart add -t freebsd-ufs -s ${BYTES}b da1
dd if=/home/user/Desktop/system.uzip of=/dev/da1p1 bs=4M status=progress

...and yes, I can boot into a graphical desktop like this. No ramdisks! No reroot! No chroot! In fact, no files on the bootable medium outside of /boot.

Here is the running system:

FreeBSD% mount
/dev/da1p1.uzip on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
tmpfs on /tmp (tmpfs, local)
tmpfs on /var/log (tmpfs, local)
tmpfs on /usr/local/etc/X11/xorg.conf.d (tmpfs, local)
tmpfs on /usr/local/var (tmpfs, local)
tmpfs on /usr/home (tmpfs, local)
tmpfs on /var (tmpfs, local)
map -hosts on /net (autofs)
<above>:/tmp on /etc (unionfs, local)

For the last line, I cheated a bit by using a little bit of unionfs because /etc apparently needs to be writable so that we can use the (wireless) network. So once the system was fully running, I ran sudo mount -t unionfs /tmp /etc. If the unionfs proves to cause stability issues, we can find some other trickery around it.

Which brings me to the overall question: How much less stable will everything become if we replace the tmpfs usage with unionfs?

Only one way to find out: Try it out. Which is now pretty easy given that we have a neat procedure for experimentation.