search

regolith-linux / voulage

A package builder
13 stars 6 forks source link

Unifying archive package structure #34

Open khos2ow opened 5 months ago

khos2ow commented 5 months ago

Current State

As of now the apt URLs for each release have hardcoded Regolith version in it and it's a bit unpredictable when it comes to upgrading to another release (Regolith, or underlying OS: Debian, Ubuntu). Currently the apt URL is as follow:

https://regolith-desktop.org/testing-debian-testing-amd64
                             |       |      |       |
                             |       |      |       * Architecture
                             |       |      * Distro Version/Codename
                             |       * Distro Name
                             * Regolith Stage

and apt source entry is as follow:

deb [arch=amd64] https://regolith-desktop.org/testing-debian-testing-amd64 testing main
                                              |       |      |       |     |       |
                                              |       |      |       |     |       * Component
                                              |       |      |       |     * Distro Version/Codename
                                              |       |      |       * Architecture
                                              |       |      * Distro Version/Codename
                                              |       * Distro Name
                                              * Regolith Stage

Proposed Change

This proposal suggests to cleanup and shorten the url with the following format:

https://archive.regolith-desktop.com/ubuntu noble main
                                     |      |     |
                                     |      |     * Component
                                     |      * Distro Version/Codename (plus potential Regolith Stage)
                                     * Distro Name

and apt source entry as follow:

deb [arch=amd64] https://archive.regolith-desktop.com/ubuntu noble main
          |                                           |      |     |
          |                                           |      |     * Component
          |                                           |      * Distro Version/Codename (plus potential Regolith Stage)
          |                                           * Distro Name
          * Architecture

Changes

  1. move Architecture out of url to deb source file entry
  2. change base domain from regolith-desktop.org to archive.regolith-desktop.com
  3. move Distro Name as the top level path of the package archive
  4. move Distro Version/Codename out to the distribution section of its own
  5. move Regolith Stage out and append it to distribution section as needed
  6. Regolith Stage will not ever have the Regolith versions (e.g. release-3_1, etc)

Note that item 2 is my personal preference and I think it looks cool to have this, but ultimately this is the only optional item in this proposal.

Examples

The followings are example of what the deb apt urls will look like:

Considerations

Backward Compatibility

The proposed solution can be backward compatible because the changes are only in the URLs and with redirect rules on the running webserver most probably can be achieved.

Multiple Versions

Multiple versions of the same package must be supported with this new proposal because Regolith Stage (i.e. Regolith stable releases: e.g. release-3_1, etc) is being moved out of the url. This raises two points:

  1. it is not currently possible to have multiple versions of the same package to be published using rerepo
    • alternative packaging tools (such as freight or aptly) most probably will solve this though
  2. is it at all needed (and supported) to have multiple versions of the same package?
    • what is the compatibility matrix of Regolith? Can 3.2 be installed on Jammy or Focal? Can 3.1 (or 3.0) be installed on Noble? etc. If the answer to these questions are no, then that ultimately means there won't be a need to have multiple packages per distro/version/package triad
khos2ow commented 5 months ago

@kgilmer @SoumyaRanjanPatnaik FYI.

I opened the issue at Voulage repo first to have a discussion and if we get to a point that we would likely wanted to move forward then it'd be great to create another discussion thread in main repo or discussions page for broader community visibility and engagements.

kgilmer commented 5 months ago

@khos2ow your proposal is fantastic! Perhaps this falls into the "multiple versions" block you have above ~ I also was unable to get reprepro to host the same package across multiple architectures. But, as a consequence of having to create a seperate debian repository for every stage/dist/codename/arch touple, we sidestep collisions in parallel builds. So, I think the deb workflow will require some changes there as well such that various builds do not compete.

freight looks cool! aptly seems a bit more than we need, upon first glance. Do you have a preference? Do you see risks?

kgilmer commented 5 months ago

Also we will still need a way of encoding the regolith stage somewhere, as it's common for users to consume from testing. Perhaps as a prefix to the host, or the first element of the path?

khos2ow commented 5 months ago

freight looks cool! aptly seems a bit more than we need, upon first glance. Do you have a preference? Do you see risks?

I also agree that Aptly might be a bit overkill for our use case and Freight seems to be exactly what we want. And deb workflow definitely needs to change, I think less on the build side, and more on the publish side.

One big difference is that, as far as I could gather, freight needs to run on the web server where the packages are hosted, as opposed to current state of building, packaging, generating deb repo, rsync everything.

khos2ow commented 5 months ago

Also we will still need a way of encoding the regolith stage somewhere, as it's common for users to consume from testing. Perhaps as a prefix to the host, or the first element of the path?

The way I'm thinking about this is point 5 and 6 from the original proposal:

  1. move Regolith Stage out and append it to distribution section as needed
  2. Regolith Stage will not ever have the Regolith versions (e.g. release-3_1, etc)

That means differentiation between stages will be right there in the deb source file, BUT, release version slug will never be visible anywhere in the deb url and/or deb source file.

And in all of the cases above, Regolith versions will just be packages (plural) inside the dist/ and pool/. At least this versioning part is theory and depends on 1) ability to use freight and such and more importantly 2) if multiple regolith version per distro release could be possible. (e.g. can we have Regolith 3.0, 3.1, and 3.2 on Noble? Or is it just 3.2 on Noble and nothing else is supported?)

kgilmer commented 5 months ago

deb [arch=amd64] https://archive.regolith-desktop.com/ubuntu noble-unstable main

:+1:

can we have Regolith 3.0, 3.1, and 3.2 on Noble? Or is it just 3.2 on Noble and nothing else is supported?

From the project start, the only target that had been supported but is no longer is Ubuntu 18.04. @SoumyaRanjanPatnaik and I recently discussed this topic and thought it best to only support the last + current LTS release, plus any current non-LTS release for Ubuntu. (I'm counting all official Debian releases as "LTS"). So, I plan to deprecate support for Bullseye and Focal in 3.2. I haven't done it yet because I want to know if something in 3.2 fails on those older platforms in beta before the release. But this is the policy, LMK what you think:

Regolith releases target the last + current LTS release, plus a latest non-LTS release for Ubuntu if exists.