[Contribution] `qvm-upgrade-template` (easy in-place upgrades for Debian and Fedora templates)

[kennethrrosen]

How to file a helpful issue

The problem you're addressing (if any)

Per official documentation, it is recommended that users download latest templates through qvm-template before "switching" everything from a pre-existing template to the newer template. qvm-prefs allows for a somewhat seamless transition of preferences. But inheritance and persistence is a bit more complicated for the non-technical user.

This makes upgrading-in-place a very attractive option for new and experienced users alike. Fedrora documentation and Debian documentation suggest this process is for the "advanced" user. This should not be the case.

The solution you'd like

Include as a qubes command the option to "upgrade-in-place" a template, perhaps qvm-upgrade-debian and qvm-upgrade-fedora. I've provided basic Bash scripts for doing so, with an option for upgrading multiple templates of the same distro at once:

For Fedora-based templates. For Debian-based templates.

At present, both scripts were tested successfully for upgrading Fedora-37 to Fedora-38 and Debian-11 to Debian-12. These scripts might also benefit from qvm-clone commands to create .bak templates and a help/usage function.

The value to a user, and who that user might be

Such scripts would lower the barrier-to-entry for non-technical users interested in adopting Qubes. Furthermore, it would lower the amount of time spent on upgrading individual templates a user may have created based on the defaults, becoming a value-add. It would also limit bandwidth on current Qubes documentation.

[kennethrrosen]

I'm willing to help further (including documentation and improving the code), but am unclear how to enlist a reviewer/mentor.

[deeplow]

We talked about this during the summit. @marmarta is the go-to person for this and she has some ideas on how to go about it. If I recall correctly the first goal is to expose the fact that there are template upgrades available in the updater and the second part would be to actually make it possible for the user to inplace upgrade them.

Exposing it via the UI is even better than a cli utility, but it'll certainly need a backend. And a qvm-template-upgrade script might be the way to go.

I think issue can be merged into #6904.

[andrewdavidwong]

I think issue can be merged into https://github.com/QubesOS/qubes-issues/issues/6904.

I'm not sure if they're really the same. #6904 seems to be about trying to help the user migrate from one customized template to another, whereas this is about scripting the in-place upgrade steps for templates. They could end up achieving a result that is close enough to count as effectively the same for most users, but there are subtle differences. Also, this is an issue for a specific package contribution. (At least, that's what appears to be so far.)

[deeplow]

So I think #6904 is talking about the user facing / front-end side of things and this one about the backend. You're right, they don't necessarily need to be the same issue, but it's important to have them tangled together since #6904 will depend on #8605 (how I wish GitHub had issue dependencies!).

[andrewdavidwong]

Well, it depends on how the developers decide to handle things. If this truly ends up being a contrib package, then I'm not sure whether the general GUI can be based on it, since contrib packages are optional and not installed by default. (Users have to opt-in.) Also, it remains to be seen whether the devs even agree with this approach as a long-term solution.

how I wish GitHub had issue dependencies!

It would be convenient, but it's not strictly necessary. We can just add the https://github.com/QubesOS/qubes-issues/labels/S%3A%20blocked label to the blocked issue and specify in text what's blocking it. This is more flexible, since sometimes no issue exists yet for the thing doing the blocking, or it's actually multiple issues or an entire project.

[marmarta]

Oh I love this! I have recently made a PR to expose "hey! this template is stale!!!" information in the GUI (in the updater widget); it would be absolutely amazing to have those script as a tool to use in the Template Manager tool. I think a qubes-template-upgrade script is a way to go, sure - I can put GUI wrappers around it, but a working CLI script is 1. really wanted by a lot of people 2. easier to wrap in GUI than GUI is to wrap around in CLI :D

[DemiMarie]

The GUI tool will want machine-readable output, including progress information, so that it can display what is going on and has happened without complex parsing. In particular, the GUI tool is likely to be run in dom0, so it needs to treat the script output as untrusted.

[kennethrrosen]

@DemiMarie @marmarta I can add JSON outputs and a command line option for that output, if that'd speed things along. Do you think JSON would be suitable, or do you have another format in mind? I've also quickly added a "help" option when no arguments are provided to the scripts. Open to further suggestions and any additional features

[marmarek]

Do you think JSON would be suitable, or do you have another format in mind?

JSON isn't really suitable for progress information (it's just a single object, not stream of updates). Progress could be reported by simply printing progress: NN (where NN is percents). But TBH, I think we can start with start with running the script in a terminal window when user selects to update a template. Proper integration with reliable progress reporting would likely require using DNF/APT API (like we do in qubes-vm-update tool, not just wrapping the commands you use currently. But lets not get into "perfect is enemy of the good".

BTW few comments to the scripts: the Fedora script creates cache block device, but doesn't use it; the Debian script opens gnome-terminal (to start the template?) but also doesn't use it. And also, avoid using sudo - it won't work on minimal template (or if somebody removes qubes-core-agent-passwordless-root package). Use qvm-run -u root instead.

[deeplow]

I took peek at the scripts and applied those improvements. Let's get this nice addition rolling :muscle:

[kennethrrosen]

Have updated the codes to correct this issue: https://github.com/QubesOS/qubes-issues/issues/8725#issuecomment-1953046377

[kennethrrosen]

May be worth merging these or adding as an option for future major releases: https://github.com/QubesOS/qubes-dist-upgrade/blob/main/scripts/upgrade-template-standalone.sh

[kennethrrosen]

per discussions with @deeplow https://github.com/QubesOS/qubes-issues/issues/8605#issuecomment-1896475484 and @marmarek https://github.com/QubesOS/qubes-issues/issues/8725 I've overhauled the scripts, merged into one script for use with both fedora- and debian-based templates and completed testing. The new repo and script can be found here https://github.com/kennethrrosen/qubes-template-upgrade/blob/07cd12f419f4715170aed3d4198683fabad4700a/qvm-template-upgrade

[ben-grande]

per discussions with @deeplow #8605 (comment) and @marmarek #8725 I've overhauled the scripts, merged into one script for use with both fedora- and debian-based templates and completed testing. The new repo and script can be found here https://github.com/kennethrrosen/qubes-template-upgrade/blob/07cd12f419f4715170aed3d4198683fabad4700a/qvm-template-upgrade

You are still using sudo. See @marmarek 's comment https://github.com/QubesOS/qubes-issues/issues/8605#issuecomment-1771378749.

On the top of the script, use set -o nounset -o errexit and handle errors if they ocurr.

New templates have the version set in their feature os-version, no need to cat /etc/fedora-release etc. This was nonexistent before but now it is the default. Download Fedora 40 template and check out its features. So maybe keep it for sometimes for compatibility reasons for templates that were not updated yet.

For the same reason, template type should not be provided by the user, but gathered via the qube feature os-distribution. Soon there will be os-distribution-like, so templates based on Fedora and Debian will also work https://github.com/QubesOS/qubes-core-agent-linux/pull/504.

User may have more than the default repositories, they need to be upgraded also, glob the /etc/apt/sources.list.d for *.sources and *.list.

[ben-grande]

For starting the qube, use --skip-if-running else it will fail. There is also no need to qvm-start as qvm-run will start the qube.

Do not sleep, handle the race conditions. Use qvm-shutdown --wait to hang the command until the qube is powered off.

[kennethrrosen]

Thanks @ben-grande ; and for all your work on "qusal." I've incorporated some of those changes https://github.com/kennethrrosen/qubes-template-upgrade/blob/main/qvm-template-upgrade but I'm not clear on how to alter the script to gather the os-distribution so that template type shouldn't have to be inputed by the user. One of the reasons the script displays the new /fedora-release or /debian_release is to ensure the user that the upgrade was done successfully. In part, this was initially created in case someone has named their template something other than the standard names.

[deeplow]

Happy to see some movement here :)

I'm not clear on how to alter the script to gather the os-distribution so that template type shouldn't have to be inputed by the user.

I'm guessing this is via qubes-features. Something like:

qvm-features fedora-40-xfce os-version

[kennethrrosen]

Then could the potential answer be have the user put their desired template name in, have the script decide the next logical distribution (based on qvm-prefs output), and the preform the upgrade on the clone?

[ben-grande]

Thanks @ben-grande ; and for all your work on "qusal."

Thanks!

I've incorporated some of those changes https://github.com/kennethrrosen/qubes-template-upgrade/blob/main/qvm-template-upgrade but I'm not clear on how to alter the script to gather the os-distribution so that template type shouldn't have to be inputed by the user. One of the reasons the script displays the new /fedora-release or /debian_release is to ensure the user that the upgrade was done successfully. In part, this was initially created in case someone has named their template something other than the standard names.

qvm-run has not --skip-if-running, qvm-start has. Please do man qvm-run and man qvm-start. Read my comment above again.

Then could the potential answer be have the user put their desired template name in, have the script decide the next logical distribution (based on qvm-prefs output), and the preform the upgrade on the clone?

Does not seem like a good idea to detect based on template name, many users have templates without the named mentioning fedora and without the version in the name fedora-40. I have many tpl-sys-pgp, use qvm-features os-distribution and os-version as instructed above instead of relying on qvm-prefs to ge the qube name.

[kennethrrosen]

I think we're on the same page. I had attempted to account for the various template names one might have. Your solution is more robust. I've updated the script to correct thenqvm-start and qvm-run mistakes and to rely only on what appears in qvm-features to determine how to go about the upgrade. I will test these changes later today.

[kennethrrosen]

I've tested the script on several templates and various naming conventions; further testers would be appreciated. But the script works as expected. Thanks again, @ben-grande for your review and pointers, and @deeplow for your gut-checks. Debian templates before debian-12 do not possess qvm-features functions like os-distribution so in order to test a Debian-11 to Debian-12 upgrade, I added a fallback which still relies on cat /etc/debian_version. This can be removed in the final version.

I'd eventually hope this gets worked into a gui, but for now I've used qvm-run -p to display the output to give users a sense of time-to-completion

Latest version is in the main branch: https://github.com/kennethrrosen/qubes-template-upgrade/tree/main

[ben-grande]

Still doesn't support Debian .sources repository configuration deb822. I saw that you supported it previously but reverted later on.

Debian supports both .sources and .list. Always look the diff of your commits to see if you removed something important and always check upstream documentation (deb822).

[kennethrrosen]

Good catch. Reverted.

[kennethrrosen]

I've also created a small GUI following QubesOS style to accompany the script for those who don't want to use the shell: https://github.com/kennethrrosen/qubes-template-upgrade/blob/main/qubes-template-upgrade-gui

[kennethrrosen]

Per contributor guidelines, I've also converted the script to Python.

[ben-grande]

• Use the functions that were imported qvm_run directly instead of the program qvm-run. Same thing for qvm_clone, qvm_start, qvm_shutdown etc.
• Run pylint in the file, there are many messages.
• get_template_vesion() can return the version in different formats, /etc/fedora-release - > string, VERSION_CODENAME -> string, /etc/debian_version -> MAJOR.MINOR. This is a problem when it is passed to next_debian_version(), which only maps to codenames.
• next_debian_version()
• Do not use ls to globs, they are fragile. Search for shellcheck issue about this.
• There is no guarantee of the existence of *.list, the same way you fix *.sources, fix the *.list existence check. Yes, it is the default, but this might change by the system in the future or by the user. Don't expect, check.
• qubes-vm-update will get many more reports as it is a more commonly used tool. Maybe borrow some code if there is something useful. And now seeing this in python, maybe it doesn't even make sense to have a separate tool to perform the same functions as vmupdate but doing a major release upgrade instead, it should be a feature of vmupdate, not a separate file that will have code duplication.
• Be careful with quotes in /etc/os-release, strip the quotes always, some distributions might insert them or not.
• Don't cat | grep, useless cat, search online for this.
• Don't return or compare against "unknown", do it against None.

[deeplow]

qubes-vm-update will get many more reports as it is a more commonly used tool. Maybe borrow some code if there is something useful. And now seeing this in python, maybe it doesn't even make sense to have a separate tool to perform the same functions as vmupdate but doing a major release upgrade instead, it should be a feature of vmupdate, not a separate file that will have code duplication.

I was taking a look at this the other day and I think its architecture pretty much fits into what we're trying to achieve. We could:

• create a vmupgrade.py, similar to vmupdate.py and making use of qube_connection and update_manager as well as the agent.
• modify the update agent to have necessary functionality to perform upgrades
• lastly create the command qubes-vm-upgrade, which is just another entrypoint for vmupgrade.py.

Advantages with this approach are many:

• code reuse
• reduction of dom0 vm output parsing

Later steps could include the integration with the GUI updater, but that is out of scope for the ticket. But it could make use of the os-eol qubes feature. This is something I'd like to contribute to.

[deeplow]

Without the ingenious gumption and go-getter attitude of @kennethrrosen, we wouldn't be here today. However, I decided to do some exploration on the feasibility of doing this instead via vmupdate (what the GUI updater uses), as Marek suggested. This branch has a very simple PoC that adds a new --version-upgrade VERSION to qubes-vm-update for the Fedora CLI.

After doing that initial work, I can confirm that ultimately this vmupdate is probably what we want to use. Here are a few reasons:

1. Distribution upgrades are already pretty much just "special" updates and almost all of the logic is there.
2. Progress reporting is already implemented
3. Parallel upgrades are basically for free
4. Integration with GUI updater will be trivial and this way it doesn't need a separate GUI (Although @kennethrrosen's is pretty nice and already enough.

I though this would be a pretty trivial (re)implementation done in a weekend. However, after spending a significant part of today attempting to add dnf_api support, I didn't get anywhere close to a fedora dist upgrade, since the API doesn't map well at all to the respective commands (kudos to @piotrbartman for the work that's already there!). So I guess I may need some help to get the APT / DNF APIs working.

What's missing?

• DNF (CLI) missing clean prior to dist-upgrade
• DNF (GUI) all is missing (my initial attempt didn't yield anything)
• APT (CLI/GUI) all is missing
• ArchLinux (we can probably defer this since there are no default templates)
• dom0 logic: clone / backup previous version (volume restore point?)
• missing automated way to know what's the next version that should go to (on Fedora this is easy but Debian, not so much)

I'm happy to spend a few more days working on this over the next month or so, if this feels like a worthy pursuit. But I do not want to suck out the contribution energy out going on here.

And I know doing with vmupdate requires a lot more setup (qubes-builderv2) and is significantly more complex (the code changes will be less, probably, but the code to understand is a lot more). Would folks be OK with this alternative or should we explore both in parallel and see which one is the most viable / suitable?

[kennethrrosen]

Would folks be OK with this alternative or should we explore both in parallel and see which one is the most viable / suitable?

This project in its current form could be a beneficial stopgap for users intimidated by the user-led upgrade-in-place procedure. So I'm happy to maintain the package as a contrib in testing while also working with @deeplow to work this into vmupdate for a much-later release.

We could also poll forum members as to whether the additional upgrader and GUI would be welcome in the meanwhile as we explore the more integrated approach.