System Update utility - Githubissues

probonopd commented 3 years ago

Write a System Update utility that installs the update into a new Boot Environment.

Along the lines of

 pkg upgrade --fetch-only -y
 beadm list
 beadm create c572e
 beadm mount c572e /media/c572e
 chroot /media/c572e
  pkg upgrade -y
  pkg upgrade -r FreeBSD -y
  exit
 beadm list
 beadm umount c572e

Thanks @grahamperrin

We could then make the next boot go into the new Boot Environment, and ask the user there whether all subsequent boots should continue to go there. Or something like that.

grahamperrin commented 3 years ago

Thanks, that's from my quick and dirty record of a first attempt https://pastebin.com/ppT7LfVT where (amongst other things) the pkg upgrade -y failed due to poudriere-related files being not found in the chroot(8) environment.

I took my hints from Upgrading from FreeBSD 9.3 to FreeBSD 10.1 using beadm and freebsd-update – Dan Langille's Other Diary (2015-03-13).

I see options such as --chroot and --rootdir in pkg(8), my first experiment with --chroot failed (through me not yet having a full understanding of the manual page). I guess that using one or both of those options in combination with (or in lieu of) chroot(8) will allow things to work with e.g. my poudriere repository, I simply haven't had time to digest the man pages.

For the pkg stuff above, the phrase System Update is potentially misleading in that base (the system) is not yet packaged. So better name the utility Software Update or something like that.

A System Update to the FreeBSD -RELEASE base of helloSystem should use freebsd-update(8).

A System Update to the FreeBSD -CURRENT base of helloSystem might be out of scope (without PkgBase).

FreeBSD upgraded, no `/dev/null` after `chroot`

grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo pkg install beadm
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        beadm: 1.3.2

Number of packages to be installed: 1

10 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching beadm-1.3.2.txz: 100%   10 KiB  10.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing beadm-1.3.2...
[1/1] Extracting beadm-1.3.2: 100%
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % pkg info -x mysql
mysql57-client-5.7.32
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % beadm list
BE      Active Mountpoint  Space Created
default NR     /            4.4G 2021-02-04 14:49
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % freebsd-version -kru
12.2-RELEASE-p3
12.2-RELEASE-p3
12.2-RELEASE-p3
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo beadm create 12.2-RELEASE-p3
Created successfully
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo beadm mount 12.2-RELEASE-p3 /media/12.2-RELEASE-p3
Mounted successfully on '/media/12.2-RELEASE-p3'
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % chroot /media/12.2-RELEASE-p3
chroot: /media/12.2-RELEASE-p3: Operation not permitted
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo chroot /media/12.2-RELEASE-p3
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # sudo pkg upgrade -y
sudo: account validation failure, is your account locked?
sudo: a password is required
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # whoami
root
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # pkg upgrade -y
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (18 candidates): 100%
Processing candidates (18 candidates): 100%
The following 16 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        abseil: 20200923.2_1 -> 20200923.3
        drm-fbsd12.0-kmod: 4.16.g20201016 -> 4.16.g20201016_1
        libunwind: 20200331_1 -> 20201110
        libx264: 0.161.3020 -> 0.161.3039
        mysql57-client: 5.7.32 -> 5.7.32_1
        opencv-core: 3.4.1_39 -> 3.4.1_40
        pciids: 20201127 -> 20201231
        protobuf: 3.13.0,1 -> 3.14.0,1
        readline: 8.0.4_1 -> 8.1.0
        sudo: 1.9.5p1 -> 1.9.5p2
        wayland: 1.18.0_4 -> 1.19.0
        xorg-server: 1.20.9_1,1 -> 1.20.9_3,1
        zstd: 1.4.5_1 -> 1.4.8

Installed packages to be REINSTALLED:
        freeglut-3.0.0_2 (direct dependency changed: libXxf86vm)
        libdvdread-6.1.0 (options changed)
        virtualbox-ose-additions-5.2.44_3 (options changed)

Number of packages to be upgraded: 13
Number of packages to be reinstalled: 3

15 MiB to be downloaded.
[1/16] Fetching zstd-1.4.8.txz: 100%  504 KiB 516.6kB/s    00:01    
[2/16] Fetching xorg-server-1.20.9_3,1.txz: 100%    1 MiB   1.5MB/s    00:01    
[3/16] Fetching wayland-1.19.0.txz: 100%  120 KiB 123.0kB/s    00:01    
[4/16] Fetching virtualbox-ose-additions-5.2.44_3.txz: 100%  682 KiB 698.7kB/s    00:01    
[5/16] Fetching sudo-1.9.5p2.txz: 100%  942 KiB 964.9kB/s    00:01    
[6/16] Fetching readline-8.1.0.txz: 100%  362 KiB 370.5kB/s    00:01    
[7/16] Fetching protobuf-3.14.0,1.txz: 100%    3 MiB   2.9MB/s    00:01    
[8/16] Fetching pciids-20201231.txz: 100%  216 KiB 220.7kB/s    00:01    
[9/16] Fetching opencv-core-3.4.1_40.txz: 100%    2 MiB   2.2MB/s    00:01    
[10/16] Fetching mysql57-client-5.7.32_1.txz: 100%    2 MiB   1.9MB/s    00:01    
[11/16] Fetching libx264-0.161.3039.txz: 100%  678 KiB 694.2kB/s    00:01    
[12/16] Fetching libunwind-20201110.txz: 100%  128 KiB 131.2kB/s    00:01    
[13/16] Fetching libdvdread-6.1.0.txz: 100%  121 KiB 123.4kB/s    00:01    
[14/16] Fetching freeglut-3.0.0_2.txz: 100%  223 KiB 228.1kB/s    00:01    
[15/16] Fetching drm-fbsd12.0-kmod-4.16.g20201016_1.txz: 100%    2 MiB   2.1MB/s    00:01    
[16/16] Fetching abseil-20200923.3.txz: 100%  798 KiB 817.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/16] Upgrading pciids from 20201127 to 20201231...
[1/16] Extracting pciids-20201231: 100%
[2/16] Upgrading readline from 8.0.4_1 to 8.1.0...
[2/16] Extracting readline-8.1.0: 100%
pkg: Cannot open /dev/null:No such file or directory
[3/16] Upgrading zstd from 1.4.5_1 to 1.4.8...
[3/16] Extracting zstd-1.4.8: 100%
pkg: Cannot open /dev/null:No such file or directory
[4/16] Upgrading wayland from 1.18.0_4 to 1.19.0...
[4/16] Extracting wayland-1.19.0: 100%
pkg: Cannot open /dev/null:No such file or directory
[5/16] Upgrading libunwind from 20200331_1 to 20201110...
[5/16] Extracting libunwind-20201110: 100%
pkg: Cannot open /dev/null:No such file or directory
[6/16] Upgrading protobuf from 3.13.0,1 to 3.14.0,1...
[6/16] Extracting protobuf-3.14.0,1: 100%
pkg: Cannot open /dev/null:No such file or directory
[7/16] Upgrading xorg-server from 1.20.9_1,1 to 1.20.9_3,1...
pkg: Cannot open /dev/null:No such file or directory
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # file /dev/null
/dev/null: cannot open `/dev/null' (No such file or directory)
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # exit
exit
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % file /dev/null
/dev/null: character special (0/19)
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo beadm umount 12.2-RELEASE-p3 
Unmounted successfully
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ %

Re: https://markmail.org/message/ediqqmqrogbv6fum#query:+page:1+mid:ediqqmqrogbv6fum+state:results I assume that

FreeBSD 12 requires an extra step to make /dev/null available in the chroot environment
FreeBSD 13 and -CURRENT include some magic, to automatically fulfil the /dev/null requirement.

probonopd commented 3 years ago

https://twitter.com/vermaden/status/1360562154836004869 describes upgrading from 12 to 13 as:

beadm create 13 beadm mount 13 chroot /var/tmp/BE-13.z4IanfZt mount -t devfs devfs /dev freebsd-update upgrade -r 13.0-BETA2 freebsd-update install freebsd-update install pkg upg freebsd-update install beadm activate 13 reboot

grahamperrin commented 3 years ago

Thanks, that's useful.

https://www.freebsd.org/cgi/man.cgi?query=pkg-alias(8) upg is not an alias. Neither does it appear here:

@vermaden please, where did you learn the upg abbreviation?

If the second run of freebsd-update install is for world, then what is the third?

grahamperrin commented 3 years ago

@probonopd ~~for an upgrade with 0.4.0 (0D26) you'll not need the devfs line. So,~~ for example:

🚫 longhand – draft – I should advise users to not yet follow this routine in its entirety (note, any yellow alert below) …

with a clean installation of either 0.4.0 (0D26) or 0.5.0 (0E4) from hello-0.5.0_0E4-FreeBSD-12.1-amd64.iso
sudo -u root sh
observe the lecture-related hint
enter the password
sed -i -e 's|release_2|latest|g' /etc/pkg/FreeBSD.conf
pkg lock hello
y
pkg install beadm && mkdir /tmp/upgrade-freebsd
beadm create 13.0-BETA2-hs-iso-115 && beadm mount 13.0-BETA2-hs-iso-115 /tmp/upgrade-freebsd && chroot /tmp/upgrade-freebsd
mount -t devfs devfs /dev
freebsd-update upgrade -r 13.0-BETA2
freebsd-update install
exit
umount /tmp/upgrade-freebsd/dev
beadm umount 13.0-BETA2-hs-iso-115 && beadm activate 13.0-BETA2-hs-iso-115
exit
for the activated boot environment to become effective, restart helloSystem
Control-Alt-F2 switch to ttyv0
login at the console – do not attempt to use the desktop environment
sudo -u root sh
freebsd-update install
pkg bootstrap -f
pkg upgrade
freebsd-update install
exit
shutdown -r now

… then ⚠ https://github.com/helloSystem/ISO/issues/136 check whether Falkon is installable

with steps very similar to what's drafted above, it is not installable – as far as I can tell, still blocked by #115 so I begin to wonder whether (below, https://github.com/helloSystem/Utilities/issues/33#issuecomment-780083496) sudo -u root sh can be a workaround (to problems that may be associated with a locked root user).

grahamperrin commented 3 years ago

https://cgit.freebsd.org/src/commit/usr.sbin/freebsd-update/freebsd-update.sh?id=48ffe56ac5b7adb5b851d32be12b2ec0f13705a4 (2006-08-31) ▶ https://cgit.freebsd.org/src/tree/usr.sbin/freebsd-update/freebsd-update.sh?id=48ffe56ac5b7adb5b851d32be12b2ec0f13705a4#n553

# Check that we are root.  All sorts of things won't work otherwise.

– and from BSD Server Maintenance - iXsystems, Inc. - Enterprise Storage & Servers (2013-09-30):

… When updating the system, type:

freebsd-update fetch; freebsd-update install

This will update the list of available software and then update the software. You must be root to apply such updates. …

– and so on.

https://cgit.freebsd.org/src/tree/usr.sbin/freebsd-update/freebsd-update.sh?id=48ffe56ac5b7adb5b851d32be12b2ec0f13705a4#n555

    echo "You must be root to run this."

– wonder why I never heard that echoed in the context of https://github.com/helloSystem/ISO/issues/115 …

freebsd-update(8) without an echo of "You must be root to run this."

https://lists.freebsd.org/pipermail/freebsd-current/2021-February/078833.html

vermaden commented 3 years ago

@vermaden please, where did you learn the upg abbreviation?

Do not remember. Seriously :)

But pkg autor works the same for autoremove.

Probably a habit after long use of IBM TSM - Tivoli Storage Manager (now known as Spectrum Protect). Their command line also needed only that much part of command that was unique.

If the second run of freebsd-update install is for world, then what is the third?

After executing freebsd-update install the freebsd-update writes this at the end of 'world' install process:

Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/usr/sbin/freebsd-update install"
again to finish installing updates.

So the third freebsd-update install is for 'removing old shared object files'.

for an upgrade with 0.4.0 (0D26) you'll not need the devfs line.

Not sure about helloSystem upgrade but when I first started 12.2 ==> 13.0 upgrade without /dev mounted in the chroot then the freebsd-update exited after installing files with information that /dev/tty is not available and its not possible to interactively 'work' on the configuration files differences like /etc/fstab or /etc/sysctl.conf files.

Regards.

grahamperrin commented 3 years ago

From https://github.com/helloSystem/ISO/issues/138#issuecomment-778615459

… root user; … Mac. …

When I last checked, updates/upgrades to Mac OS X (familiarly macOS) involved booting an environment that does allow the root user.

So:

if (with helloSystem denial of root) https://github.com/helloSystem/Utilities/issues/33#issuecomment-779647953 above can not usefully involve chroot(8)
then maybe First Aid https://github.com/helloSystem/hello/issues/90#issuecomment-779701997

– however goals such as First Aid and Storage utilities might be mid- to long-term.

IMHO a more immediate priority should be to:

allow users of FreeBSD to update/upgrade the system using normal methods; and
allow these methods to be as close as possible to frictionless.

Thanks

(To do: link to a non-official shortlist of vulnerabilities.)

grahamperrin commented 3 years ago

https://github.com/helloSystem/Utilities/issues/33#issuecomment-779700378 thanks @vermaden

`/dev`

~~Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.~~

Old files

That reminds me,

root@mowa219-gjp4-8570p:~ # cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old
>>> Removing old files (only deletes safe to delete libs)
/usr/sbin/fmtree
/usr/lib/debug/usr/sbin/fmtree.debug
/usr/share/man/man8/fmtree.8.gz
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
root@mowa219-gjp4-8570p:/usr/src #

– and

I do not make delete-old-libs
(off-topic from helloSystem) I can't easily recall what I have installed from ports
(OT) I do update (from source) my jail for poudriere, but I have never given thought to deletion of old files within the jail (does it take care of itself?) … maybe saner to simply create from source.

grahamperrin commented 3 years ago

https://github.com/helloSystem/docs/pull/17

grahamperrin commented 3 years ago

Taking a hint from https://pastebin.com/vSPb8Pxg (thanks to Remilia in freenode):

sudo -u root sh

– I'm trying, repeatedly, to fix the routine at https://github.com/helloSystem/Utilities/issues/33#issuecomment-779647953 above to work with root-locked helloSystem. So far, without success :-(

probonopd commented 3 years ago

We could also take an entirely different approach to "updating", e.g., installing from a new Live ISO while keeping parts of the old system around, similar to what the Mac does... or use an A/B partitioning scheme, similar to what ChromeOS does, or...

grahamperrin commented 3 years ago

… installing from a new Live ISO while keeping parts of the old system …

The parts gained from an ISO might as effectively be gained with a PkgBase approach. I think first of @igalic https://alpha.pkgbase.live/

probonopd commented 3 years ago

Not really: Integration and end-to-end tests can be run on an ISO, but not on an upgraded system. At least not easily.

grahamperrin commented 3 years ago

Integration

Way beyond me but 👍

end-to-end tests

Is an intention, to verify (with checksums) that what's installed is as it should be?

grahamperrin commented 3 years ago

@vermaden

https://github.com/helloSystem/Utilities/issues/33#issuecomment-779711271 my bad,

Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.

That was, not taking into account the chroot step. So, yes, mount -t devfs devfs /dev is a necessary next step.

vermaden commented 3 years ago

#33 (comment) thanks @vermaden

`/dev`

Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.

Old files

That reminds me,

root@mowa219-gjp4-8570p:~ # cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old
>>> Removing old files (only deletes safe to delete libs)
/usr/sbin/fmtree
/usr/lib/debug/usr/sbin/fmtree.debug
/usr/share/man/man8/fmtree.8.gz
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
root@mowa219-gjp4-8570p:/usr/src #

– and

* I **do not** `make delete-old-libs`

* (off-topic from helloSystem) I can't easily recall what I have installed from ports

* (OT) I do _update_ (from source) my jail for poudriere, but I have never given thought to deletion of old files within the jail (does it take care of itself?) … maybe saner to simply _create_ from source.

Yes, this is that step.

grahamperrin commented 3 years ago

@vermaden FYI https://pastebin.com/crR6WFCG – first observed whilst testing an upgrade with helloSystem, it's reproducible with a clean installation of FreeBSD ~~12.1-STABLE~~ 12.1-RELEASE (without helloSystem) as a starting point.

If I (exit the chroot environment etc. and) restart the system before the second run of freebsd-update install then there are no such bad system calls (and /lib/libmd.so.6 is not empty).

igalic commented 3 years ago

just one small note re beadm: if you replace it with bectl(8), which is in base, that's one less thing to install

vermaden commented 3 years ago

@vermaden FYI https://pastebin.com/crR6WFCG – first observed whilst testing an upgrade with helloSystem, it's reproducible with a clean installation of FreeBSD 12.1-STABLE (without helloSystem as a starting point.

If I (exit the chroot environment etc. and) restart the system before the second run of freebsd-update install then there are no such bad system calls (and /lib/libmd.so.6 is not empty).

If moving from 12.x to 13.x its best (and probably stated in the FreeBSD Handbook) to first upgrade 12.1 to 12.2 (or other latest 12.x version at the moment) and then do the major upgrade to 13.x.

Regards.

probonopd commented 3 years ago

The more I think about it, I come to the conclusion that I am really more in favor of a "clean install" at least for major revisions/releases that would work roughly like this:

User downloads the latest ISO
In a running helloSystem, user double-clicks the ISO
User double-clicks "Install FreeBSD", the usual installer opens
The disk from which helloSystem is currently booted is shown
If the user selects that disk, then the following happens:
- A new jail/chroot is created that is also a boot environment
- helloSystem is installed to that jail/chroot
- /home is the same for the old and the new boot environment (as far as I remember this is even the default)
- /Applications may need some special treatment (tbd)
The new boot environment is selected for the next boot (only)
Only if we reach the full graphical desktop in the next boot, then set the new boot environment permanently

Advantages:

"Clean install" = predictable results
ISO can be tested beforehand
More familiar process to Mac users, but using ZFS power

For minor (security) updates, we could still use a more traditional approach.

What do you think?

vermaden commented 3 years ago

Agreed.

grahamperrin commented 3 years ago

Thanks,

https://github.com/helloSystem/Utilities/issues/33#issuecomment-780522191

… first upgrade 12.1 to 12.2 (or other latest 12.x version at the moment) and then do the major upgrade to 13.x. …

☑ that's sane,

… probably stated in the FreeBSD Handbook) …

– unless I'm missing something, it's not.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-upgrade

grahamperrin commented 3 years ago

https://github.com/helloSystem/Utilities/issues/33#issuecomment-778611679 @vermaden with the chroot approach, there is (I think) an extremely slim possibility of shooting oneself in the foot, in cases where the system actively instructs the user to check for updates to FreeBSD.

These three frames are from a screen recording of a clean test system that originated with non-patched FreeBSD 12.1-RELEASE:

2021-02-18 00:43:53 frame

2021-02-18 00:44:18 frame

2021-02-18 00:45:06 frame

there's the instruction – run freebsd-udpate fetch.

In this chroot context, I did know to ignore the instruction. I chose to run the command – in a disposable VirtualBox session – solely to capture a little of the wrongness. (I did not install what was fetched.)

Regards

vermaden commented 3 years ago

… probably stated in the FreeBSD Handbook) …

– unless I'm missing something, it's not.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-upgrade

Then I probably seen it on FreeBSD Forums as a general advice ...

vermaden commented 3 years ago

In this chroot context, I did know to ignore the instruction. I chose to run the command – in a disposable VirtualBox session – solely to capture a little of the wrongness. (I did not install what was fetched.)

Regards

Sometimes the third 'install' step is not needed. I wish the freebsd-update would tell you that instead of telling you to 'fetch' instead.

The good part is that when you type that 'fetch' command it will not find new updates.

You may as well submit a FreeBSD BUG for it - to make freebsd-update clarified when to run 'install' 2 times and when 3 times.

Here: https://freebsd.org/send-pr.html

Hope that helps.

grahamperrin commented 3 years ago

Thanks,

… The good part is that when you type that 'fetch' command it will not find new updates. …

The third image from my previous comment:

2021-02-18 00:45:06 frame

Later frames from the same screen recording:

321 files fetched
if I'm not mistaken, it's preparing to remove files from a 12.1-RELEASE environment when truly, that environment (albeit, kernel not yet running) is 12.2-RELEASE-p3

… and preparing to update 12.1-RELEASE files, which longer exist in that state, to 12.1-RELEASE-p13, which would be inferior to the upgraded files that are already in place …

probonopd commented 3 years ago

@pkgdemon shared some ideas on IRC:

after i saw your presentation and you mentioned delta updates i realized you could turn off a bunch of base things you wouldn't even want like bsdconfig, sendmail, all kinds of things and save hundreds of megabytes for the iso. not only that you could sandbox things installed by pkg into a jail think of like having brew on macos but with freebsd pkg. assuming your primary mechanism becomes a full repositiry of appimages or something that suceeds ports with something better. it would prevent a user from doing pkg install gnome3 in the sandbox because x11 apps in a sandbox you know a great idea or won't be easy to run a full desktop like that. but if you don't care about that flexability and would rather prevent foot shooting it might be an interesting approach to think of jails and base.txz in a jail. (...) each update is like a snapshot pulled down with something like zsync. a boot environment is made, snapshot is sent to specific dataset for the OS later. reboot to appy the update. remove pkg altogether. replace with appimages. if someone wants pkg, they open jail.app and it fetches and builds a container containing the full base.txz and allows them to have a playground without continating or footshooting your system. it's a bit big brother to gate like that in a way so i don't know if it would fit your vision though. i was thinkgin you could for your build system generate your own very stripped down base.txz with only what is critical to hellosystem.

probonopd commented 3 years ago

Upgrade strategy idea: "Side-loading FreeBSD versions using Boot Environments" Only that we would use a new ISO instead of base.txz, kernel.txz, and similar https://people.freebsd.org/~dch/posts/2021-02-23-sideloading-freebsd.html

probonopd commented 3 years ago

"Updating/upgrading" is complex, requires a lot of effort on our side and may still be fragile, so I have a strong preference toward "image-based installs" and/or "clean installs" rather than traditional "package-based system updates".

Hence I envision a helloSystem update/upgrade to be much more like an Android or iOS update/upgrade than a FreeBSD update/upgrade.

Unless someone volunteers to put in significant work to package all of helloSystem, including e.g., making packages of all packages for which we have even the slilghtest configuration changes, and addresses all the concerns below.

But let's think through it:

We need to think about

Binary compatibility. Unlike "typical" FreeBSD users, many helloSystem users are non-technical and value binary compatibility highly
Robustness. Unlike "typical" FreeBSD users, many helloSystem users are non-technical and cannot "quickly fix up small issues" or answer any questions concerning technical decisions duing the update process
Testability, Reproducability, and Supportability. Unlike "typical" FreeBSD users, many helloSystem users are non-technical and hence will need from other people should any issues occur. Hence it is very important to keep the test matrix (combination of FreeBSD base version and packages version) as small as possible, and allow each supported combination to be easily tested/reproduced, e.g., by using a Live ISO

When we talk about "updating/upgrading", we need to consider at least four different actions:

Patch the installed FreeBSD minor version (~weekly)
Upgrade the packages repository (~quarterly)
Upgrade to a new FreeBSD minor version (~yeary)
Upgrade to a new FreeBSD major version (~bi-annually)

Each of which shall be discussed below.

Patch the installed FreeBSD minor version

# Make ZFS Boot Environment

# sudo -A -E bectl create "$(uname -v) $(date '+%Y-%m-%d-%H%M%S')" # NOTE_ Do NOT do this. While it creates a Boot Environment, the system cannot boot into it, presumably due to some characters that are not allowed for booting, such as spaces

sudo -A -E bectl create pre-update # This works and can be booted into

# Non-interactively install binary patches for FreeBSD base system
env PAGER=cat sudo -A -E freebsd-update fetch --not-running-from-cron
freebsd-update install

Tested on a FreeBSD 12.1 system, worked without any issues. We may want to enable this.

Upgrade the packages repository

We don't want latest packages because this means an unpredictable system because it is a moving target with daily updates of everything that might break anything at any time.

We also don't want quarterly packages because even there, packages are updated throughout the quarter, and as a result entire packages (e.g., browsers) may be missing entirely, making it impossible to build helloSystem ISOs. (Note: It seems like the GhostBSD project creates their own packages in part because "the quarterly repo was not as stable and reliable as expected".)

Hence, we are using the set of packages that was latest when the last FreeBSD ISO was released, known as release_0, release_1,... etc.

At some point users may want to switch to newer packages.

# Lock critical packages
sudo -A -E pkg lock hello

# Edit release_X in /etc/pkg/FreeBSD.conf

# Update all packages from the new repository metadata
sudo -A -E pkg update -f

The result would be a combination of FreeBSD base system and packages that has never been tested by the helloSystem team and cannot be reproduced by using any of our Live ISOs. Which is practically unsupportable and hence undesirable. __We may want to establish a policy that for each minor version of FreeBSD, the user should remain on the release_X that was the most recent at the time when the FreeBSD minor version was released.__

The risk is not hypotetical - https://github.com/helloSystem/ISO/issues/14 shows that one can easily end up with a broken system.

Upgrade to a new FreeBSD minor version

If we want to also upgrade minor FreeBSD versions, e.g., from 12.1 to 12.2,... then the resulting system is no longer in a consistent tested state but since each dot releale is supported only for a very short time (one year?) it might be advisable to find a way to allow for such updates.

# Lock critical packages
sudo -A -E pkg lock hello
pkg lock -l
sudo -A -E freebsd-update -r 10.2-RELEASE upgrade

https://globalengineer.wordpress.com/2017/02/15/freebsd-minor-version-upgrades says:

I had made some modifications to /etc/ssh/ssh_config on my baseline system, so rather than just overwriting the file, FreeBSD is offering me the opportunity to look at a diff of my original and the new version that FreeBSD needs to install

Since helloSystem makes quite some modifications to the configuration files, including to rc scripts, I imagine that we will run into a lot of trouble there unless we can find a way to tell freebsd-update to NEVER touch any of the modified files, and NEVER ask the user any questions. But in this case, e.g., updates in the rc scripts would not arrive on helloSystem. Which may be undesirable.

We must absolutely ensure that non-technical users cannot get stuck with a non-working system when doing this. And there is a real risk, as

https://github.com/helloSystem/docs/pull/17/files#diff-777f2fd130a3b9c2fa13129bc2aeb1f9d24b8ed81c53c516bb4b467a2201ec77R31-R38

shows.

Hence, do a clean install of new version into a mounted new ZFS Boot Environment, and then reboot into it?

Upgrade to a new FreeBSD major version

For major FreeBSD upgrades we want to only allow clean installs (every two years or so) because a lot in the helloSystem-specific parts and configuration might have changed in the meantime.

Do a clean install of new version into a mounted new ZFS Boot Environment, and then reboot into it?

Packaging all of helloSystem

In case we really want to enable package-based update scenarios that span not only FreeBSD and its packages but also helloSystem, we would need to rework the ISO generation process significantly; especially build.sh and script.hello would need to be refactored to generate packages rather than installing/modifying things in the system image directly.

Some of the existing FreeBSD packages and/or the base would need to be re-created as well, e.g., to have customizations e.g., in /etc/rc and /etc/rc.shutdown which we are currently editing "on the fly":

https://github.com/helloSystem/ISO/blob/41650b7970dc90a6da7ff9e6b18a7af530ff8d62/settings/script.hello#L238-L267

Basically we would need to find a way to get rid of script.hello](https://github.com/helloSystem/ISO/blob/experimental/settings/script.hello)...

grahamperrin commented 3 years ago

The risk is not hypotetical - helloSystem/ISO#14 shows that one can easily end up with a broken system.

Not a good example, because it's a mixture of issues in a single issue. The essence of what's there is the naming conflict; this is a helloSystem design flaw, not a problem with upgrading.

grahamperrin commented 3 years ago

a real risk, as https://github.com/helloSystem/docs/pull/17/files#diff-777f2fd130a3b9c2fa13129bc2aeb1f9d24b8ed81c53c516bb4b467a2201ec77R31-R38 shows.

Again, not a good example.

probonopd commented 3 years ago

The essence of what's there is the naming conflict; this is a helloSystem design flaw, not a problem with upgrading.

True, but it serves as an example that one can end up in situations where the Live ISO works but trouble arises once users begin to update it. In this particular instance the fix is easy, but will it be in all similar cases?

grahamperrin commented 3 years ago

will it be in all similar cases?

Yes.

With an installed system (a norm), if a boot environment is troubled:

activate then boot the previous environment.

If a boot environment can not be created (read-only ISO files, and the like):

reboot.

probonopd commented 3 years ago

If a boot environment is troubled, activate then boot the previous environment.

For this I have an idea actually:

After the upgrade, set the bootloader automatically to boot into the new BE once
When the new BE is fully booted to the desktop, set the bootloader automatically to boot this BE always

crees commented 3 years ago

Most of our modifications can be done in a robust way, that won't be broken on upgrade. For example, the modification to /etc/rc is not necessary, as we can use a boot splash screen instead using the splash kernel module. We can use any bmp image as a splash, so even a Hello image might be fun!

If we don't touch any files in /etc (and I don't think that we do apart from those), then upgrades should be seamless.

If you suggest a nice splash image I can modify loader.conf to show that, and we can dispense with the modifications to rc.

probonopd commented 3 years ago

This, white on a black background, not too large, should probably do: https://github.com/helloSystem/hello/blob/master/branding/hello_variation.svg

Please be aware that no text must be shown duing the entire boot process (after the bootloader). Not a single character.

probonopd commented 3 years ago

Doing echo "y" | env PAGER=cat sudo -A -E freebsd-update -r 12.2-RELEASE upgrade now on my beloved 12.1 system. Let's see how it goes.

I have made a BE and commented out beastie_disable="YES" in /boot/loader.conf just in case. Fingers crossed...

Actually, I would have liked to install the upgrade into a mounted (but not booted) BE but I did not know how to do that.

Now comes the showstopper for our "mere mortals" users:

The following file could not be merged automatically: /etc/group
Press Enter to edit this file in /usr/bin/vi and resolve the conflicts
manually...

Actually, it is a showstopper for me as well.

What does it want from me? Please leave my carefully edited config files untouched, ever.

What does this mean?

<<<<<<< current version
video:*:44:user,foo,demo,demo1,test3,deleteme,deleteme2,deleteme3
=======
>>>>>>> 10.2-RELEASE

Looks like it wants to delete my video group? Can this result in me not being able to see a graphical environment again?

Worst if all, I am now dropped into vi which for me is one of those pieces of software that work so differently than how I think (as someone grown up with Macs) that it makes me want to quit. How?

And how does one tell it "leave it as is"?

Same for /etc/newsyslog.conf. At this point I have enough and kill the process. Just leave my carefully edited files alone!

grahamperrin commented 3 years ago

…freebsd-update -r 10.2-RELEASE upgrade

Should probably have been:

…freebsd-update upgrade -r 12.2-RELEASE

(You don't want a downgrade from end of life 12.1 to even less lively 10.2.)

probonopd commented 3 years ago

Typo indeed.

In the course of doing this, I also found out that my nice and shiny BE I had made called FreeBSD 12.1-RELEASE-p13 GENERIC 2021-03-20-192857 turned out to be unbootable. bectl should not allow one to use names for BEs that cannot be booted!

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254441

Also I can't seem to be able to destroy it anymore:

% sudo bectl destroy "FreeBSD 12.1-RELEASE-p13 GENERIC  2021-03-20-192857"
Passwort: 
bectl destroy: leaving origin 'zroot/ROOT/pre-upgrade@2021-03-20-19:28:44-0' intact
cannot destroy 'zroot/ROOT/FreeBSD 12.1-RELEASE-p13 GENERIC  2021-03-20-192857@2021-03-20-19:28:59-0': dataset already exists
unknown error

By sheer luck I was able to boot back into the default BE but imagine that one would not have worked anymore. Would have been locked out of the machine!

crees commented 3 years ago

Freebsd-update merges new changes into /etc, which should mean that they work properly with the new base system. Thankfully, we don't merge /etc/rc.d any more (that just gets overwritten), but it is still the most tedious part of the merging process.

If you don't touch files in /etc (and to be honest, apart from /etc/rc and /etc/rc.shutdown, I don't think we do), the merge is completely automatic.

The really annoying thing is when a new user/group is added to base, and you then have to merge master.passwd and group. I guess the long term answer would be to change upgrades to use pw to add users/groups... but that's a way down the line.

I'll give some thought to how we could handle this.

probonopd commented 3 years ago

If you don't touch files in /etc (and to be honest, apart from /etc/rc and /etc/rc.shutdown, I don't think we do)

Actually we do heavily:

It should just leave these files alone entirely.

grahamperrin commented 3 years ago

https://github.com/helloSystem/Utilities/issues/33#issuecomment-803440465

…edit this file in /usr/bin/vi…

What was the shell at the time?

probonopd commented 3 years ago

/usr/local/bin/zsh I guess.

grahamperrin commented 3 years ago

For the root user?

helloSystem / Utilities

System Update utility #33

See also

FreeBSD upgraded, no `/dev/null` after `chroot`

freebsd-update(8) without an echo of "You must be root to run this."

`/dev`

Old files

`/dev`

Old files

helloSystem / Utilities

System Update utility #33

See also

FreeBSD upgraded, no /dev/null after chroot

freebsd-update(8) without an echo of "You must be root to run this."

/dev

Old files

/dev

Old files

FreeBSD upgraded, no `/dev/null` after `chroot`

`/dev`

`/dev`