search

LukeShortCloud / winesapOS

winesapOS - Game with Linux anywhere, no installation required!
GNU General Public License v3.0
854 stars 31 forks source link

Create a generic Arch Linux upgrade script #711

Open LukeShortCloud opened 7 months ago

LukeShortCloud commented 7 months ago

Our existing winesapOS upgrade script is basically an Arch Linux upgrade script that addresses issues brought up on archlinux.org . It has some extra logic to handle winesapOS configurations and packages. I would like to refactor our upgrade script into a new project simply called arch-linux-upgrade (or archupgrade to mirror the archinstall project name that exists) to handle automated and unattended upgrades. Then winesapOS can re-use that script and add on our own logic for things very specific to our project.

LukeShortCloud commented 6 months ago

We should have a list of Arch Linux versions that we can use to reference container tags for testing upgrades via a CI/CD pipeline.

https://hub.docker.com/_/archlinux/tags

A list to start off with could be the winesapOS versions that used Arch Linux (probably skipping the minor releases):

https://github.com/LukeShortCloud/winesapOS/releases

Alternatively, we could simply target every 1 year, 6 months, or 3 months.

LukeShortCloud commented 6 months ago

Goals:

Non-goals:

LukeShortCloud commented 6 months ago

Packages from https://archive.archlinux.org/ can be pre-installed into old Arch Linux containers for CI testing purposes.

LukeShortCloud commented 4 months ago

Pacman has an undocumented feature to automatically replace packages without confirmation:

$ sudo pacman -S <PACKAGE> --ask 4

https://unix.stackexchange.com/questions/274727/how-to-force-pacman-to-answer-yes-to-all-questions/584001#584001

LukeShortCloud commented 3 months ago

Arch Linux maintainers recommend using pacutils for automating Pacman operations (and not using pacman itself).

https://gitlab.archlinux.org/pacman/pacman/-/issues/102 https://github.com/andrewgregory/pacutils

LukeShortCloud commented 3 months ago

If using a pre- and post-hooks approach with archupgrade to integrate with winesapOS, we should move away from having upgrades organized for specific winesapOS versions. We have already had to reorganize upgrades many times in the past. Depending on what needs to be upgraded, it may need to happen or after any given X.Y.Z version. For example, saying "Running 3.0.1 to 3.1.0 upgrades..." is not entirely accurate since we do various repository modifications required for newer versions of winesapOS.

LukeShortCloud commented 3 months ago

We should come up with a list of Arch Linux based distributions we would want to support. Obviously winesapOS but I have heard that folks from ChimeraOS are interested (for their users who insist on turning off the read-only frzr file system). There is also ArcoLinux, Batocera, EndeavourOS, Garuda Linux, etc.

LukeShortCloud commented 3 months ago

@Thijzer @soredake @wallentx @ohaiibuzzle @Sebazzz @lucid-cloud-dreams @Frostswing @Equ1no0x @Haarolean @maxiride @DKRacingFan @Rapou7 @99powerbreaker @QushyQushy @rkr87 @AHandheldFan @A-mak88 @FabioLolix @mjkl-gh @liberodark @massatt212 @Penguin766 @KaiH-0

Are any of you interested in helping to make an archupgrade project? Similar to how there is a standardized archinsall project?

If I tagged you, that means you have helped contribute to winesapOS in a meaningful way and I am very grateful for that! In winesapOS, we have an upgrade script that fully automates Arch Linux upgrades. Instead of keeping this for winesapOS only, I want to take the experiences/learnings from that and create a new project that will work across a wider range of Arch Linux based distributions. With a fresh start and fresh new perspectives from the community, we can design this better and can create a better experience for end-users.

Why is this needed? Upgrading Arch Linux usually requires manual intervention and it is not always intuitive on how to solve it. Especially for new Linux users. I found a lot of people use Discover from KDE Plasma to do system upgrades. This upgrades one package a time. If the upgrade fails for some reason, it is left in a half broken state. I have also seen other users wait 1-2 years before upgrading which leads to various issues. Dealing with GPG keyrings is especially a big problem on older systems. KDE Plasma 5 to 6 upgrades were also horrible to deal with. This new upgrade script/program would handle everything automatically with no user interaction required.

Some questions we should answer:

I wrote down various ideas earlier in this thread related to a few of those questions that we can talk about more.

Where would we discuss this? GitHub is too slow for communication. Let's use Discord instead.

April 20th Saturday at 4:00 P.M. (16:00) UTC (Reach out to me on Discord @LukeShortCloud for the server invite) in the #misc-utilities channel

Thanks for your time! I hope to see ya'll online!

massatt212 commented 3 months ago

You should use Manjaro KDE since it's simple to install, and it will replicate SteamDeck os closer, could always make a repository that have your custom kernel and mesa drivers, or U can make your own arch installer with a script to install steamos, cause one of the problems I hers is sddm and gdm switching back and fort from game mode to desktop.

LukeShortCloud commented 3 months ago

@massatt212 We have started work on that idea already. :-) Users who insist on having an installer can use Manjaro and then our conversion script. It is very basic right now and does not cover Gamescope session yet but we are looking into expanding it. We can include our repository, tweaks, system packages, etc.

https://github.com/LukeShortCloud/winesapOS/tree/4.0.0?tab=readme-ov-file#convert-to-winesapos

LukeShortCloud commented 3 months ago

In its simplest form, archupgrade will be able to automatically address upgrade concerns from the Arch Linux news feed.

For example:

Thijzer commented 3 months ago

Hi @LukeShortCloud, I'm definitely willing to help. Although I kinda shifted to ublue atomic images for most of my machines, I still like those arch based image builds as a base as you have more control of how you package..

ohaiibuzzle commented 3 months ago

I had a couple of ideas reading this. Noting the best one I thought of down so I won't forget it later. Might be absurd, or might very well be viable winesapOS-upgrade.pdf

LukeShortCloud commented 3 months ago

@ohaiibuzzle Nice PDF! That really helps to clarify your vision and seems doable.

winesapOS 5.0.0 may include an optional read-only image and I will definitely need your help with that! We will still offer "legacy" images with the traditional writable file system for users who want and expect that. Short-term, my friends at ChimeraOS have made frzr to address this problem by shipping Btrfs volume updates. Long-term, bootc is the future by allowing us to use a Dockerfile to build the operating system file system. Directories such as /etc and /home remain writable and are kept during upgrades. Updates are delivered from a container registry. @Thijzer Universal Blue uses containers today (in a slightly different way that is RPM distro specific) and we would take a very similar approach. I spoke with a few members of their team and they will also be moving to bootc (which happens to be distro agnostic) eventually. Thanks for your continued interest and help!

archupgrade will be focused on traditional Arch Linux upgrades. Some folks have even suggested that perhaps archupgrade should help users on ChimeraOS or SteamOS switch to a writable file system. We should follow-up this winesapOS read-only discussion in the near future!

Thijzer commented 3 months ago

I completely agree with your comment @LukeShortCloud. I was thinking of picking up the work of chimera os built-tools and working out a foundation of good image building and easy image deployment. Also the layering of the cake like ublue images is what I would want in these tools. I'll read up on bootc and see if it aligns. Thanks!

ohaiibuzzle commented 3 months ago

Yeah, I would much prefer, for an Arch system anyway, for them to upgrade in an imaging-based manner rather than doing traditional package upgrade if we can. It would be much more predictable than trying to deal with potentially dicey package upgrades, since winesap is expected to just be used with Flatpaks and not having users managing packages themselves.

I could think of a few ideas to achieve package-based upgrades, I'll see which one could work and make a similar writeup if I do

LukeShortCloud commented 3 months ago

@Thijzer Do you have a Discord? You can add me @LukeShortCloud and I will send you an invite to the Discord server. We will be meeting April 20th Saturday at 4:00 P.M. (16:00) UTC. We will be focused on the archupgrade conversation there but, in the coming weeks or months, we can also have a follow-up discussion about a read-only variant of winesapOS.

lucid-cloud-dreams commented 3 months ago

@LukeShortCloud I feel really honored to be invited to this project :) I have to say, that I am not a programmer by any means! But I would say about myself, that I'm a good enough sysadmin ;) But I still want to join the call at 4pm UTC today. I am really excited how I can help the project! :)

LukeShortCloud commented 3 months ago

@lucid-cloud-dreams Your system administrator experience would be great to help provide feedback on how we tackle upgrades! Reach out to me on Discord @LukeShortCloud and I will get you an invite to our Discord server. You can join us in the #misc-utils channel in an hour when the meeting starts.

LukeShortCloud commented 3 months ago

@ohaiibuzzle Thanks for sharing your idea about a potential YAML layout (pasted below)! You mentioned this was inspired by Debian. This would be great for helping define upgrade recipes for winesapOS, ChimeraOS, etc.

phases:
  first:
    preupgrade:
      - some_bash_commands && do_something
    packages:
      - package_a: https://archive.archlinux.org/p/package_a.tar.zst
      - package_b: https://archive.archlinux.org/p/package_b.tar.zst
    postupgrade:
      - some_more_bash_commands && do_some_more_things
  second:
    preupgrade:
      - some_bash_commands && do_something
    packages:
      - package_c: https://archive.archlinux.org/p/package_c.tar.zst
      - package_d: https://archive.archlinux.org/p/package_d.tar.zst
    postupgrade:
      - some_more_bash_commands && do_some_more_things
ohaiibuzzle commented 3 months ago

I'll make a writeup similar to the other, imaging-based upgrading strategy when I get back to my desktop later, expanding on the idea, would be more detailed since we have a direction to go now

wallentx commented 2 months ago

I intended to reply to this, but I just started a new job a few weeks ago, and my focus was pulled away. I'm sorry I missed the boat! I actually have a lot of opinions about this, since it sounds like this is what the Steam Deck is doing. If you all need a hand on any ongoing/future work, I'd love to help!

LukeShortCloud commented 2 months ago

Hey @wallentx , I know you already read through everything on Discord but I'll give a quick update for everyone.

There are two goals we are targeting with archupgrade:

  1. Upgrade to the latest Arch Linux. Deal with common issues.
  2. Convert to different profiles. For example: gaming_desktop (winesapOS) or gaming_console (ChimeraOS).

With the conversion feature, distros can use archupgrade to take a minimal Arch Linux container and get a fully functional operating system. Users can either use this program on bare metal, distrobox/toolbox containers, or in a build system (ex., for bootc). Everything is declarative and state-driven.

For the programming language, most folks are leaning towards Rust but other proposed languages include Go and Bash.

@ohaiibuzzle Could you upload the latest PDF of your outline here on this GitHub Issue? That showcases the goals and non-goals plus the YAML configuration we are thinking about using.

ohaiibuzzle commented 2 months ago

Here is my notes on how I see it could be implemented, the "specification" refers to the file that contains what's explained by Luke as the "profile". "Implementation" here is the actual archupgrade binary, which may/can be implemented differently using different toolchains, so long as it fits the definition.

archupgrade.pdf

LukeShortCloud commented 1 month ago

The official Arch Linux Package Manager (alpm) library has bindings for the following languages:

https://wiki.archlinux.org/title/Alpm_based_tools

LukeShortCloud commented 6 days ago

As a quick update: this whole year I've been traveling 2 weeks every month for work. I'm also ramping up on Rust by creating an unrelated project. I'm still aiming to start work on arch-upgrade later this year once I get more free time and my Rust skills mature.

GuestSneezeOSDev commented 6 days ago

i could help but what do you mean exactly by "Arch Linux upgrade script"

GuestSneezeOSDev commented 6 days ago

if you mean a custom package manager that would be too complex for me

LukeShortCloud commented 6 days ago

@GuestSneezeOSDev All of the details can be found here: https://github.com/LukeShortCloud/winesapOS/issues/711#issuecomment-2064212817 Basically, we want to create a fully automated and non-interactive upgrade script for Arch Linux. One that can deal with common issues (outdated GPG keys, new packages from a distro, package conflicts as posted on archlinux.org, etc.).

GuestSneezeOSDev commented 6 days ago

archupgrade will be a script like archinstall which will be able to update GPG Keys update software and update the system sound good i could probably help but i do not know how to make a KDIALOG Bash script