probonopd commented 3 years ago

Write a System Update utility that installs the update into a new Boot Environment.

Along the lines of

 pkg upgrade --fetch-only -y
 beadm list
 beadm create c572e
 beadm mount c572e /media/c572e
 chroot /media/c572e
  pkg upgrade -y
  pkg upgrade -r FreeBSD -y
  exit
 beadm list
 beadm umount c572e

Thanks @grahamperrin

We could then make the next boot go into the new Boot Environment, and ask the user there whether all subsequent boots should continue to go there. Or something like that.

probonopd commented 3 years ago

echo "y" | env PAGER=cat sudo -A -E freebsd-update -r 12.2-RELEASE upgrade was the command I typed into QTerminal as a normal user.

crees commented 3 years ago

If you don't touch files in /etc (and to be honest, apart from /etc/rc and /etc/rc.shutdown, I don't think we do)

Actually we do heavily:
* https://github.com/helloSystem/ISO/tree/experimental/overlays/uzip/furybsd-settings/files/etc

* https://github.com/helloSystem/ISO/blob/experimental/overlays/uzip/hello/files/etc/rc.d/load_acpi

Most of these are extra files, which are indeed left alone.

It should just leave these files alone entirely.

It does :)

I think the issue comes only with master.passwd and group. I imagine the rc files are overwritten too, so you'll lose your splash screen.

probonopd commented 3 years ago

I imagine the rc files are overwritten too, so you'll lose your splash screen.

What would be "the right way" to do this if we indeed want those files to be modified?

grahamperrin commented 3 years ago

https://github.com/helloSystem/Utilities/issues/33#issuecomment-803440465

… 12.2-RELEASE … into a mounted (but not booted) BE …

Back to https://github.com/helloSystem/Utilities/issues/33#issuecomment-778611679
essentially, 12.2-RELEASE in lieu of 13.0-BETA2

Be aware of https://vermaden.wordpress.com/2021/02/23/upgrade-freebsd-with-zfs-boot-environments/, which offers more detail than the tweet.

Whilst the mount + chroot approach is streamlined and smart, for now I strongly suggest:

1. create, activate then boot a boot environment before beginning to upgrade the environment
1. refrain from using mount or chroot
1. take the FreeBSD Handbook approach, which includes a restart of the OS.

For learning purposes, the longhand approach may help to visualise the normal sequence of things.

If you like, run freebsd-version -kru before and after each significant step in (iii). Certainly before and after each restart of the OS.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-security-patches for updating and patching FreeBSD, e.g. 12.1-RELEASE to 12.1-RELEASE-p13.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-upgrade for upgrading FreeBSD, e.g. 12.1-RELEASE-p13 to 12.2-RELEASE-p4 at the time of writing. The Handbook example:

You'll want:

freebsd-update upgrade -r 12.2-RELEASE

– or freebsd-update -r 12.2-RELEASE upgrade, although the first form seems more commonplace (see for example http://issue.freebsdfoundation.org/publication/?i=501780&p=8&pp=1&view=issueViewer and the highlighted line at https://diigo.com/0k172h).

probonopd commented 3 years ago

So based on my earlier results (n=1), updating is not dangerous while upgrading is.

grahamperrin commented 3 years ago

Do you mean danger of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254441?

probonopd commented 3 years ago

No, I mean danger of being asked questions. Which my users certainly won't understand.

grahamperrin commented 3 years ago

This issue is somewhat messy.

Whilst I'm reasonably happy to test and offer guidance about commands and routines where interest is shown, ~~I don't recall suggesting freebsd-update(8) as a basis for a System Update/Upgrade utility.~~

Postscript: correcting myself, from https://github.com/helloSystem/Utilities/issues/33#issuecomment-761946911:

… A System Update to the FreeBSD -RELEASE base of helloSystem should use freebsd-update(8). …

I mean, freebsd-update(8) is the norm for binary updates and binary upgrades to supported versions of RELEASE, with (as far as I know) no existing alternative that is simpler.

users

Does this exclude advanced users, who are (in the first paragraph of the User Guide) invited to try the system?

probonopd commented 3 years ago

I don't recall suggesting freebsd-update(8) as a basis for a System Update/Upgrade utility

Are there other (better) methods for upgrading from 12.1 to 12.2?

users

The non-technical "creative makers" and switchers that helloSystem is primarily intended for.

probonopd commented 3 years ago

Here is yet another way in which updating/upgrading can break our preconfiguration: https://github.com/helloSystem/ISO/issues/195#issuecomment-813219233

Debian seems to have dpkg divert which has been suggested for this kind of issue but I don't thnk pkg has it?

igalic commented 3 years ago

I'm not sure i fully understand the issues, but would pkg register fit the bill? https://dan.langille.org/2021/02/16/preventing-a-given-package-from-being-installed/

probonopd commented 3 years ago

@igalic what we need here is to prevent one file from being installed, not one package.

Your link may help in another situation tough:

https://github.com/helloSystem/ISO/blob/372c83eb6e6d179da366adb22c4e02ba68e2e520/settings/script.hello#L360-L363

kettle-7 commented 3 years ago

What if the base system was a git repository mounted ro and you mounted a separate zfs subvolume for rw folders (like /Users), and in an update you'd just run

mount -o rw,remount / # is remount even necessary on BSD?
cd /
git pull
cd -
mount -o ro /

probonopd commented 3 years ago

I don't think that git is a good match for managing gigabytes of binaries. Plus, git does ask the user questions in the case of merge conflicts.

kettle-7 commented 3 years ago

There's always ostree, it's like git but endless os uses it.

probonopd commented 3 years ago

Definitely unwelcome technology. Not human-friendly. Cryptic GUID-like names of files. Not "manageable" by hand. Not simple.

kettle-7 commented 3 years ago

The other thing I can think of is SystemImageKit and SystemExtensionKit

probonopd commented 3 years ago

SystemImageKit is Linux/GRUB only.

probonopd commented 3 years ago

tl;dr

This is mainly a summary for my future self:

FreeBSD itself

FreeBSD isn't sufficiently described with versions like "12.2". You can have "12.2" and it is still outdated
You also need to look at the number that follows RELEASE-.... For me it helps to think of 12.2-RELEASE-p7 as 12.2.7
In fact, the kernel and the FreeBSD "userland" can have different versions. freebsd-version -ku can be used to show the kernel version and the userland version
Note that "userland" here does not mean anything installed using Ports or Packages
To update FreeBSD "itself" (not: ports and packages) in one go without asking questions one can use sudo env PAGER=cat freebsd-update fetch install and then reboot

Thanks to @grahamperrin for being patient enough explaining it to me as long as was needed for me to finally get it.

(still to be found out whether we can prevent this from overwriting our carefully crafted custom config files such as modified rc scripts)

Packages

sudo env PAGER=cat pkg.real update
sudo env PAGER=cat pkg.real upgrade --yes

(still to be found out how to prevent them from overwriting our carefully crafted custom config files)

grahamperrin commented 3 years ago

… still to be found out how to update those without them asking questions, …

https://github.com/helloSystem/hello/issues/161#issuecomment-944930048 showed:

csh

setenv PAGER cat …

… (Still a bit unclear) sudo env PAGER=cat freebsd-update fetch install may need to be run again after rebooting …

Not required. The advice to run the command was because you did not mention restarting the system.

probonopd commented 3 years ago

(moved to https://github.com/helloSystem/hello/issues/161#issuecomment-944944341)

grahamperrin commented 3 years ago

Better to continue package-related discussion under https://github.com/helloSystem/hello/issues/161

probonopd commented 3 years ago

Indeed.

probonopd commented 3 years ago

For updating packages, we could probably re-use most of the code from our application installer GUI. It already knows how to display a grapical progress bar for pkg, and allows more advanced users to look at the command line output.

probonopd commented 3 years ago

Create a new BE prior to doing anything (TODO: only if FreeBSD and/or package updates are available).

Then, update FreeBSD:

Next, update packages:

Anyone volunteers to test?

probonopd commented 3 years ago

Should appear in the "Developer Preview" folder in the next continuous build, 0G25.

probonopd commented 3 years ago

Known issues

computer:///...drive items appear (items for unmounted drives). gvfs to blame?
Everything that is contained in FreeBSD packages (but has been modified by helloSystem) gets overwritten. We need to find a way to tell pkg to leave certain files alone. Examples:
- /usr/local/share/slim/themes/default/background.jpg (wallpaper)
- /usr/local/sbin/automount (automounter; we use a much newer version than is in packages
- Files that we forcefully deleted (because we absolutely don't want them on the system but many packages depend on them) re-appear. Examples:
- /usr/local/share/fonts/dejavu/
- /usr/local/share/fonts/Liberation/

Essentially much of what script.hello does in terms of customizing the system gets undone when one updates packages.

We'd probably have to rethink (=make more complicated) the way we put the system together if we really want end users to update the system using packages.

In the meantime, sudo pkg lock --yes automount slim gvfs dejavu liberation-fonts-ttf) can somewhat work around this. (There may be more packages that need to be locked as well.)

probonopd commented 2 years ago

sudo env PAGER=cat freebsd-update upgrade -r 13.1-RELEASE does not silence the questions; still getting

The following components of FreeBSD seem to be installed: kernel/generic src/src world/base world/lib32

The following components of FreeBSD do not seem to be installed: kernel/generic-dbg world/base-dbg world/lib32-dbg

Does this look reasonable (y/n)?

Would env EDITOR=cat work? Trying sudo env EDITOR=nano PAGER=cat freebsd-update upgrade -r 13.1-RELEASE but having no questions asked at all is what we really need.

probonopd commented 2 years ago

My real life experience.

Thought it would be nice to run the latest version of PrusaSlicer.

Wouldn't it be awesome if this could be done without having to change half the operating system?

FreeBSD% sudo pkg  install PrusaSlicer
Passwort: 
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 45 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        FreeCAD: 0.19.2_3
        frei0r-plugins: 1.7.0.18
        frei0r-plugins-opencv: 1.7.0.18
        gdal: 3.3.3_2
        opencv: 4.5.3_4
        shotcut: 21.03.21_3

New packages to be INSTALLED:
        abseil: 20211102.0 [FreeBSD]
        gcc11: 11.3.0 [FreeBSD]
        libixion: 0.17.0_1 [FreeBSD]
        py39-boost-libs: 1.79.0_1 [FreeBSD]
        py39-numpy: 1.22.4_1,1 [FreeBSD]
        py39-setuptools: 62.1.0_1 [FreeBSD]

Installed packages to be UPGRADED:
        PrusaSlicer: 2.3.3 -> 2.3.3_4 [FreeBSD]
        boost-libs: 1.72.0_6 -> 1.79.0_1 [FreeBSD]
        botan2: 2.19.1 -> 2.19.2 [FreeBSD]
        codeblocks: 20.03_1 -> 20.03_2 [FreeBSD]
        firebird25-client: 2.5.9_1 -> 2.5.9_2 [FreeBSD]
        firefox: 95.0.2_1,2 -> 105.0.1,2 [FreeBSD]
        gnustep-base: 1.27.0_2 -> 1.27.0_6 [FreeBSD]
        gnustep-gui: 0.28.0_2 -> 0.28.0_3 [FreeBSD]
        harfbuzz-icu: 3.2.0 -> 4.4.1_1 [FreeBSD]
        icu: 70.1_1,1 -> 71.1,1 [FreeBSD]
        libcdr01: 0.1.7_1 -> 0.1.7_2 [FreeBSD]
        libcmis: 0.5.2_2 -> 0.5.2_5 [FreeBSD]
        libe-book: 0.1.3_21 -> 0.1.3_24 [FreeBSD]
        libmspub01: 0.1.4_19 -> 0.1.4_20 [FreeBSD]
        liborcus: 0.16.1_1 -> 0.17.2_1 [FreeBSD]
        libqxp: 0.0.0_19 -> 0.0.0_20 [FreeBSD]
        libreoffice: 7.2.5.2 -> 7.3.5.2 [FreeBSD]
        libvisio01: 0.1.7_5 -> 0.1.7_8 [FreeBSD]
        libzmf: 0.0.2_24 -> 0.0.2_25 [FreeBSD]
        openjfx14: 14.0.2.1+1_4 -> 14.0.2.1+1_7 [FreeBSD]
        openvdb: 8.1.0_5 -> 9.1.0 [FreeBSD]
        poppler: 21.12.0 -> 22.06.0 [FreeBSD]
        poppler-glib: 21.12.0 -> 22.06.0 [FreeBSD]
        poppler-qt5: 21.12.0 -> 22.06.0 [FreeBSD]
        poppler-utils: 21.12.0 -> 22.06.0 [FreeBSD]
        qt5-core: 5.15.2p263 -> 5.15.5p165_2 [FreeBSD]
        qt5-webkit: 5.212.0.a4_7 -> 5.212.0.a4_10 [FreeBSD]
        raptor2: 2.0.15_19 -> 2.0.15_22 [FreeBSD]
        sfcgal: 1.4.0 -> 1.4.1_1 [FreeBSD]
        spidermonkey78: 78.9.0_4 -> 78.15.0_2 [FreeBSD]
        tesseract: 5.1.0 -> 5.1.0_1 [FreeBSD]
        thunderbird: 91.9.1 -> 102.3.1_1 [FreeBSD]
        webkit2-gtk3: 2.34.2 -> 2.34.6_3 [FreeBSD]

Number of packages to be removed: 6
Number of packages to be installed: 6
Number of packages to be upgraded: 33

The process will require 46 MiB more space.
535 MiB to be downloaded.

Proceed with this action? [y/N]:

Chose to sudo pkg upgrade instead. At first it said it would uninstall nothing, so I started to do it.

But then, after having downloaded and installed a bunch of stuff, it said the following. Had I known it would end like this, I would not even have attempted the upgrade.

Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 874 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        drm-fbsd13-kmod: 5.4.144.g20211013
        e2fsprogs-libss: 1.46.4
        hdf-szip: 2.1.1
        p7zip: 16.02_3
        py38-cairo: 1.18.1_2,1
        py38-gobject3: 3.38.0
        py38-markdown: 3.3.4
        py38-matplotlib: 3.4.1
        py38-mutagen: 1.42.0_2
        py38-pygments: 2.7.2
        py38-pyliblo: 0.10.0_2
        py38-pyside2: 5.15.2_2
        py38-qt5-core: 5.15.4_2
        py38-qt5-dbus: 5.15.4_2
        py38-qt5-gui: 5.15.4_1
        py38-qt5-multimedia: 5.15.4_3
        py38-qt5-network: 5.15.4_2
        py38-qt5-printsupport: 5.15.4_3
        py38-qt5-qml: 5.15.4_3
        py38-qt5-svg: 5.15.4_3
        py38-qt5-webchannel: 5.15.4_2
        py38-qt5-webengine: 5.15.4_3
        py38-qt5-widgets: 5.15.4_3
        py38-rdflib: 5.0.0
        py38-shiboken2: 5.15.2_2
        py38-sip: 5.5.0_1,1

Grudgingly accepted. Will probably have to install the py39 equivalents by hand. Mere mortals would be totally lost here. This is why I think package managers are tools for developers, but not for non-technical end users.

sudo pkg install e2fsprogs-libss hdf-szip p7zip py39-cairo py39-gobject3 py39-markdown py39-matplotlib py39-mutagen py39-pygments py39-pyliblo py39-pyside2 py39-qt5-core py39-qt5-dbus py39-qt5-gui py39-qt5-multimedia py39-qt5-network py39-qt5-printsupport py39-qt5-qml py39-qt5-svg py39-qt5-webchannel py39-qt5-webengine py39-qt5-widgets py39-rdflib py39-shiboken2 py39-sip

pkg: No packages available to install matching 'e2fsprogs-libss' have been found in the repositories
pkg: No packages available to install matching 'hdf-szip' have been found in the repositories
pkg: No packages available to install matching 'p7zip' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-core' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-dbus' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-gui' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-multimedia' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-network' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-printsupport' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-qml' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-svg' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-webchannel' have been found in the repositories
pkg: No packages available to install matching 'py39-qt5-widgets' have been found in the repositories

Bummer!

Total breakage. Now what?

sudo pkg install devel/py-qt5

After a reboot, I cannot open many applications anymore, getting

This application requires at least version FBSD_1.7 of libc.so.7 to run.

Please update it and try again.

So far I've been running FreeBSD 13.0-RELEASE. So doing

sudo freebsd-update upgrade -r 13.1-RELEASE

takes forever, hanging on Preparing to download files... for a looong time with no progress indication. Then got Fetching 18519 patches.

The following file could not be merged automatically: /etc/group
Press Enter to edit this file in /usr/bin/vi and resolve the conflicts
manually...

Absolute nightmare! I don't even know how to get out of vi, much less how to edit anything using it. Please just leave my files alone!

Typing the cryptic :q! a couple of times gets me out of vi.

Stuff like

+<<<<<<< current version
 video:*:44:user
+=======
+video:*:44:
+realtime:*:47:
+idletime:*:48:
+>>>>>>> 13.1-RELEASE

absolutely makes me fear that Xorg will be broken.

So Does this look reasonable (y/n)? n, retry using sudo env PAGER=cat freebsd-update fetch install.

Does sudo env PAGER=cat freebsd-update upgrade -r 13.1-RELEASE similarly work? No.

Trying sudo env EDITOR=nano PAGER=cat freebsd-update upgrade -r 13.1-RELEASE next.

Had to know a lot about the system to manually edit the files in nano.

Then:

To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".
FreeBSD% sudo /usr/sbin/freebsd-update install
Passwort: 
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

After having done that, I now seem to have successfully moved from 13.0 to 13.1, but it is not something a non-technical user could have done.

probonopd commented 2 years ago

EDITOR=edit is even better because it comes with FreeBSD base.txz.

grahamperrin commented 2 years ago

… nightmare! … vi, …

⚙ D36786 freebsd-update.sh: EDITOR: if not set, prefer ee

grahamperrin commented 2 years ago

… Would env EDITOR=cat work? …

No. cat(1) is not an editor.

grahamperrin commented 2 years ago

… PrusaSlicer …

Not part of the operating system (not integral to FreeBSD).

… Total breakage. Now what? …

Boot the boot environment that preceded the breakage.

Questions about pkg(8), and other things that are not part of the operating system, will be easier to answer if asked as questions in Q&A: https://github.com/helloSystem/hello/discussions/categories/q-a

grahamperrin commented 2 years ago

… Stuff like

+<<<<<<< current version
 video:*:44:user
+=======
+video:*:44:
+realtime:*:47:
+idletime:*:48:
+>>>>>>> 13.1-RELEASE

…

Think of that as you would think of a merge with Git. From Git - Advanced Merging:

probonopd commented 2 years ago

The purpose of this thread is to show which challenges we have to overcome in oder to arrive at a no-questions-asked upgrade process that we could integrate in the System Update utility.

I fear this would have to be solved by FreeBSD proper. In the meantime, package-based system upgrades are not suitable for non-technical users.

grahamperrin commented 2 years ago

~~NB, my earlier comment:~~

~~… I don't recall suggesting freebsd-update(8) as a basis for a System Update/Upgrade utility. …~~

freebsd-update(8) is, essentially, the norm for binary updates and binary upgrades to supported versions of RELEASE.

I thought that you already decided, long ago, to not use freebsd-update in the context of an update utility for helloSystem.

… package-based system upgrades …

Do you mean PkgBase (for the operating system, FreeBSD)? Or pkg-upgrade(8) (for packages of ports)?

grahamperrin commented 2 years ago

https://github.com/helloSystem/Utilities/issues/33#issuecomment-1266665324

… package-based system upgrades are not suitable for non-technical users.

Maybe true from the helloSystem perspective; not true as a sweeping statement.

From https://www.ghostbsd.org/about for GhostBSD, which takes pkg-based approaches:

… a graphical application to install software and update your system. …

… We try to simplify FreeBSD to lower the entry-level of using FreeBSD on a desktop or laptop. We provide all the benefits of the FreeBSD operating system benefits combined with our in-house GUI tool. …

From https://github.com/helloSystem/Utilities/issues/33#issuecomment-761946911 (2021-01-18, the first response to the opening post):