Trying to catch up on the Arch Linux Port, Ilia is Incompatible With Tracker3 3.5.2 (tracker-sparql-3.0)

[gardotd426]

So, at first I was getting meson configure errors because I didn't have tracker-sparql 2.0 libraries. I checked my tracker3 install, and turns out it's been updated on Arch to 3.0.

I don't know how long ago this happened, but I used to be able to build and use ilia, but now it's incompatible (usually you can query pacman with pacman -F /usr/lib/libtracker-sparql-2.0.so and it will tell you the last version of tracker3 (or any package) to carry it.

Anyway, with 3.0, ninja errors out with the following:

[1/18] Compiling Vala source src/Config.vala ../src/DialogPage.vala ../...age.vala ../src/tracker/TrackerPage.vala ../src/windows/WindowPage.vala
FAILED: src/ilia.p/Config.c src/ilia.p/DialogPage.c src/ilia.p/DialogWindow.c src/ilia.p/Main.c src/ilia.p/IconLoader.c src/ilia.p/SessionController.c src/ilia.p/apps/DesktopAppPage.c src/ilia.p/commands/CommandPage.c src/ilia.p/keybindings/ConfigParser.c src/ilia.p/keybindings/I3Ipc.c src/ilia.p/keybindings/KeybindingsPage.c src/ilia.p/notifications/RoficationClient.c src/ilia.p/notifications/RoficationPage.c src/ilia.p/textlist/TextListPage.c src/ilia.p/tracker/TrackerPage.c src/ilia.p/windows/WindowPage.c
valac -C --debug --debug --pkg x11 --pkg pango --pkg gtk+-3.0 --pkg gobject-2.0 --pkg glib-2.0 --pkg gio-2.0 --pkg gee-0.8 --pkg gdk-pixbuf-2.0 --pkg gdk-3.0 --pkg cairo-gobject --pkg atk --pkg tracker-sparql-3.0 --pkg json-glib-1.0 --pkg gio-unix-2.0 --color=always --directory src/ilia.p --basedir ../src src/Config.vala ../src/DialogPage.vala ../src/DialogWindow.vala ../src/Main.vala ../src/IconLoader.vala ../src/SessionController.vala ../src/apps/DesktopAppPage.vala ../src/commands/CommandPage.vala ../src/keybindings/ConfigParser.vala ../src/keybindings/I3Ipc.vala ../src/keybindings/KeybindingsPage.vala ../src/notifications/RoficationClient.vala ../src/notifications/RoficationPage.vala ../src/textlist/TextListPage.vala ../src/tracker/TrackerPage.vala ../src/windows/WindowPage.vala
../src/tracker/TrackerPage.vala:156.34-156.62: error: The name `get' does not exist in the context of `Tracker.Sparql.Connection' (tracker-sparql-3.0)
  156 |                 var connection = Tracker.Sparql.Connection.get ();
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compilation failed: 1 error(s), 0 warning(s)
ninja: build stopped: subcommand failed.

[kgilmer]

Hi @gardotd426 , it's great to hear from you. Hope all is well. Regarding the issue, what branch of ilia are you building? How best can I reproduce this build issue myself? (not being an Arch user, is there some cloud image I could run in a VM to quickly get to the repro?) TIA

[gardotd426]

Welp. I went to go look into it a bit to try and find the easiest way for you to repro this, and I've run into something....

How are you even able to build this? libtracker-sparql-2.0-dev, which is a full-on requirement for this package according to the debian/control file, hasn't been available on Ubuntu since 20.04, Ubuntu is also on libtracker-sparql-3.0 since for 22.04, 22.10, and 23.04. Aren't you guys (Regolith Linux) on 22.04 now?

Anyway, that would be the easiest way to reproduce it then, is just try to build it on Ubuntu 22.04 or later.

If you want my methodology, I grepped sparql recursively in the project root directory, and replaced all the instances of 2.0/2_0/2-0 with 3.0/3_0/3-0. There are 3 relevant ones for me, but you might also wanna replace the debian/control instance as well if you plan to build it with dpkg or something. git clone https://github.com/regolith-linux/ilia.git; cd ilia; grep -ir sparql . will get you each instance.

[kgilmer]

ilia builds from multiple branches per GNOME version (the upstream dependency relating to tracker). For more recent versions of Debian and Ubuntu, this branch is used: https://github.com/regolith-linux/ilia/tree/ubuntu/jammy

The easiest way to determine what branch may being used is to look in the package model files from which Regolith is built. For example, the most recent version (lunar), specifies a branch for ilia: https://github.com/regolith-linux/voulage/blob/main/stage/unstable/ubuntu/lunar/package-model.json. Note these JSON files are merged, from outward to inside. So the set of packages common to all versions in the unstable environment is this guy: https://github.com/regolith-linux/voulage/blob/main/stage/unstable/package-model.json

[gardotd426]

ilia builds from multiple branches per GNOME version (the upstream dependency relating to tracker). For more recent versions of Debian and Ubuntu, this branch is used: https://github.com/regolith-linux/ilia/tree/ubuntu/jammy

Sweet! Building succeeds now.

Note these JSON files are merged, from outward to inside. So the set of packages common to all versions in the unstable environment is this guy: https://github.com/regolith-linux/voulage/blob/main/stage/unstable/package-model.json

Yeah, see that's the problem, I originally was like "well, r2_2 is probably somewhat frozen, so I'll try main/master. But the same two build failures I had on r2_2 (ilia and regolith-control-center) are still unchanged on main, they have the same problems.

regolith/1%43.0-1 for regolith-control-center builds properly, so I guess I'll stick to lunar for now. I have everything I need to build for the PKGBUILD building correctly now (luckily I don't need most of the packages in manifest.txt because they are already available on Arch), so now the only thing is to completely rewrite the 1.6 PKGBUILD (which still works to this day, I've been maintaining it, just not moving to 2.X). Which is going to be a whole monster, the current PKGBUILD is like 800 lines, and I can't keep any of it. I have to completely re-write a new few-hundred lines, and though it'll be less lines, since I'm moving to a more upstreamed approach where I directly clone and build the source repos instead of extracting .debs and copying them*, it's a lot more thought required than just ar x-ing a bunch of stuff and patching a few files.

*I always mostly stuck to .deb extraction because before 2.0 pretty much none of the packages even HAD GH repos, so it wasn't even an option. But now that it is, I definitely want to go that route.

[gardotd426]

I'll go ahead and close. Thanks, Ken.

[kgilmer]

@gardotd426 regarding your package build setup, if you're interested in working more closely together, I could setup a github workflow for arch builds if you think it's something you could work with. It would be based from the current debian builder, but I'd stub out the sections for source and binary package builds. I'd look into setting up some kind of arch cloud image on my local proxmox setup, and tie that into the Arch github workflow. Essentially you'd have bash functions as hooks into the build system to generate packages. I don't know if there is a place for arch packages or you must host them yourself, but either way we should be able to get something working. OTOH if you prefer to continue with your existing setup, totally understand. LMK if you'd like me to setup the initial arch build integration in voulage.

[gardotd426]

I actually had already been thinking about how nice it would be if Voulage worked with Arch in some way shape or form, I just didn't really know how it could be adapted to that because I'm not a big brain developer like you lol, so I hesitated to suggest it.

I'd stub out the sections for source and binary package builds. I'd look into setting up some kind of arch cloud image on my local proxmox setup, and tie that into the Arch github workflow.

Yeah, and you could also just install EndeavourOS, Garuda, or ArcoLinux which are all basically "Vanilla Arch or mostly Vanilla Arch with a GUI installer" since upstream Arch has a CLI installation method (it has a new archinstall script that's run from the CLI but I've only ever used the manual method so I can't vouch for it). Endeavour or Arco would be best, they just use Calamares. It would greatly reduce onboarding time.

Essentially you'd have bash functions as hooks into the build system to generate packages.

If you could help me with the bash scripting that sounds perfect, since it would theoretically only have to be done once and then it could be mildly modified if, for example, you add a new application like ilia or something.

I don't know if there is a place for arch packages or you must host them yourself, but either way we should be able to get something working.

So, the way Arch does packaging is similar to Debuntu in some ways and dissimilar in other ways. So, the end binary packages that we install with pacman are .pkg.tar.zst archives, same as how .debs are ar archives, and they just contain the files to be installed in the same tree structure as .debs. When you run a pacman update, your system grabs these .pkg.tar.zst archives from the repositories and installs them. pacman -U file.pkg.tar.zst is the same as dpkg -i file.deb.

However, this package set obviously isn't in the main repositories (for now), it's in the AUR (Arch User Repository). We have programs like yay and paru that use the same syntax as pacman. For example, pacman -S gimp installs gimp, yay -S regolith-i3xrocks installs i3xrocks from the AUR.

The way AUR installations are done, is there is an AUR git repository that contains the PKGBUILD file and any necessary patches or other files that require bundling in. PKGBUILDs are scripts, and when you run makepkg -s from the directory containing the PKGBUILD, it installs any makedeps, pulls in the sources, compiles/extracts the program and installs it into a tree structure in a tmp directory, then packages it into a pkg.tar.zst file.

I'll upload the PKGBUILD for the regolith pkgbase to a gist. Here you go. Yeah it's long, but it's actually ALL the Regolith packages I ship, as opposed to just one. You can do individual PKGBUILDs or when it makes sense you can do pkgbase PKGBUILDs, like for Regolith or for the amdgpu-pro drivers.

The PKGBUILD there has one of most types of build - some source builds using python, some source builds using make, some extracting .deb builds (a lot of AUR packages just extract .deb packages), etc. Skimming that should give you an idea for how it is done. I'm not saying I expect you to write any of the PKGBUILD, I'm just showing you this since you said you weren't sure how stuff is packaged and I'm assuming you'll want to know what I need.

Basically, if I could have a way to easily clone and build all the packages I need, that would make things MUCH easier. Hell, I could even just have voulage as a make dependency and build everything with voulage inside the PKGBUILD, and ship the whole thing. Or I can keep the packages individual (each package_pkgname () section is its own package and ends up with its own pkg.tar.zst).

If you want, I can get you a list of the packages I don't need to ship. Or you can find them yourself once you install Endeavour or something, because yay and paru allow you to search for packages in the repos and AUR by just running yay <searchterm>/paru <searchterm>. But it'd probably be easier for me to just give you a list, because it might get confusing, because I need to limit makedeps and dependencies to repo packages, because not everyone uses AUR helpers like yay and paru, they just clone the AUR repo and run makepkg -si, and makepkg cannot pull in AUR dependencies. I limit AUR packages to the optional dependencies array.

I know this is a rather large brain dump, but I've been using Arch for 5 years so feel free to ask me anything you need clarification on, and also if you wanna maybe flesh out what you're thinking as far as Arch CI-ing goes.

Oh, and just in case the regolith PKGBUILD is a bit daunting, I also maintain the electronplayer AUR package, and it's a much smaller PKGBUILD with only one package, so I'll upload that gist if you want to look at it as well. Looking at that will tell you how I package source builds, and if you want to look at how archived builds go (like .deb extraction) you can just look at one of the regolith packages. electronplayer PKGBUILD.

The reason I'd prefer to stick to source-building is twofold:

1. Some packages need to be built against the user's libraries. For example a lot of python stuff has problems if it's compiled on a different subversion, and it's ALWAYS gotta be recompiled when moving major versions, like 3.10 to 3.11.
2. The .debs went away once for a lot of packages (well they didn't go away, but I couldn't find the 2.X versions of a lot of the new stuff), and this would make me less dependent on anything other than the upstream source code.

If you're talking about bash hooks, idk how feasible it is, but maybe we could have functions (like a better version of my extract_deb function in the regolith PKGBUILD), one for meson/ninja, one for make, one for python. Like build_meson build_gnu and build_py or something. And then we could just tie the list of packages to their git repos and relevant branches and do build_meson $_ilia or something. I can include the script as part of the AUR git repository. Or we could use voulage and just add an Arch function or something, honestly I think the logistics of how best to integrate it are best left up to you, I'm just trying to give you the information you might need to make the best decision.

[kgilmer]

That's a lot of info, thanks! A few big questions that come to mind:

1. how are machine architectures handled? In debian, there is no cross-compiler setup. You have to build packages on the architectures you support. For Regolith since we support amd64 and arm64 it means I need at least two build hosts. How does arch AUR handle this?
2. I believe Arch has a "rolling release" setup. Ubuntu and Debian have named, periodic releases. These are represented in voulage with the CODENAME value. Eg ubuntu-focal or debian-bookworm, codename is the name after the distro name. But, I'm also building Debian Testing,which is in a sense like a rolling release in that there is never a release. To deal with this I simply do not specify a "release" target for that distro-codename combo. All users must then consume packages from the "testing" stage. Put another way, debian-bullseye is in the release stage but debian-testing will never be, by it's nature. This works but I'm not sure it's ideal. Any thoughts about how to model Arch distro in voulage?

[gardotd426]

how are machine architectures handled? In debian, there is no cross-compiler setup. You have to build packages on the architectures you support. For Regolith since we support amd64 and arm64 it means I need at least two build hosts. How does arch AUR handle this?

In the PKGBUILD. If you look at the PKGBUILD I put up for you to peruse you'll see arch=('x86_64') next to each package_<pkgname> function. That determines whether someone on ARM can try to install it. The reason I did that was because I was extracting .deb packages which are architecture-specific (unless they specify otherwise). But if we are building everything from source (except for any theme/icon packs/i3xrocks modules I need to include which are arch-independent**), I can put arch=('any') and it will compile on either x86_64 or ARM assuming the code is functional and compiles correctly, and since you support ARM then it should do that easily.

I believe Arch has a "rolling release" setup. Ubuntu and Debian have named, periodic releases. These are represented in voulage with the CODENAME value. Eg ubuntu-focal or debian-bookworm, codename is the name after the distro name. But, I'm also building Debian Testing,which is in a sense like a rolling release in that there is never a release. To deal with this I simply do not specify a "release" target for that distro-codename combo. All users must then consume packages from the "testing" stage. Put another way, debian-bullseye is in the release stage but debian-testing will never be, by it's nature. This works but I'm not sure it's ideal. Any thoughts about how to model Arch distro in voulage?

Yep. That's fine. Arch uses arch-rolling in lsb-release and whatnot. So you can just use that.

** I forgot to mention in my last comment that a few of the packages I ship don't even need to compile since they're just copy-and-pasting files, like any themes/icons/i3xrocks modules. But I can just handle those on my own, it's really not difficult. They already come how they need to be, I can just copy them into the packaging directory in the PKGBUILD, no problem.

To further explain point 1, you can run yay -Si <packagename> and get the package information (regardless of whether it's in the AUR or official repos). Here's an AUR package that supportsany` architecture:

Name            : electron-nightly-npm
Version         : 26.0.0.nightly.20230525-1
Description     : Build cross platform desktop apps with web technologies
Architecture    : any
URL             : https://electronjs.org/
Licenses        : MIT  custom
Groups          : None
Provides        : electron  electron-nightly
Depends On      : ffmpeg  gtk3
Optional Deps   : None
Conflicts With  : None
Replaces        : None
Download Size   : 88.17 MiB
Installed Size  : 233.76 MiB
Packager        : Nico Jensch <dr460nf1r3@chaotic.cx>
Build Date      : Sun 28 May 2023 04:45:53 AM EDT
Validated By    : MD5 Sum  SHA-256 Sum  Signature

So, that's what we'll do. Just put any in the PKGBUILD. It shouldn't need anything on your side, because I just have to put any in the PKGBUILD in the arch= arrays and that's it.

[kgilmer]

Semi-related: https://github.com/regolith-linux/regolith-desktop-website/pull/22

[gardotd426]

Oh great, a large overhaul! lol but that will at least give me a chance to get it ready alongside the updates here (I wanted to do that with the move from 1.X to 2.0, but it wasn't nearly as public back then so it was hard). So that's pretty cool.

To generalize across X11 and Wayland sessions, Xresource keys have been renamed to remove "i3". This means that users with Xresource overrides on earlier versions of Regolith will need to update the key names. Specifically, keys starting with i3-wm now start with wm. For example, in Regolith 2.x to change gaps size, i3-wm.gaps.inner.size is being renamed to wm.gaps.inner.size.

I'm assuming a simple sed -i 's/i3-//g' ~/.Xresources-regolith or whatever will suffice for this?

[SoumyaRanjanPatnaik]

Really excited about this. Arch was my go to distribution before I switched to regolith as my daily driver.

[gardotd426]

Really excited about this. Arch was my go to distribution before I switched to regolith as my daily driver.

1.6 has been in the AUR for a year (make sure you remove --class=floating_window from the gnome-control-center parts of the config but otherwise should be fine). But yeah I've been hoping to get this back up to speed, the changes from 1.6 to 2.0 made extracting .debs like the old packages (and a ton of other AUR packages also) impossible, and it was very cumbersome to try and move over to source for everything but this is a real good start.

[kgilmer]

I'm assuming a simple sed -i 's/i3-//g' ~/.Xresources-regolith or whatever will suffice for this?

Yes, I believe it's entirely regular

While there are luls in 3.0 release work, I'll work on getting the scaffolding for an arch package (repo?) builder. :rocket:

[gardotd426]

Some update:

I've finally reduced the 1.6 PKGBASE from like 20-something packages to 7, I haven't eliminated anything (the move to 2.X will obviously require adding a few packages instead, like ilia and regolith-control-center). But what I did was, every i3xrocks block used to be inside its own package, like the PKGBUILD had 20-something subpackages, regolith-i3 which handles the i3-related stuff (I obviously don't build i3 because i3 is already in the Arch repos), remontoire-regolith, regolith-styles, regolith-i3xrocks, regolith-st (because of the included patches), regolith-desktop-config, and then one package each for all the blocks, like i3xrocks-info, i3xrocks-focused-window-name, etc.

Now all the i3xrocks blocks are all just one package because I'm just cloning the regolith-i3xrocks-config repo and moving the files into my regolith-i3xrocks-config package. It took me an evening to figure out how to remove all the i3xrocks-<blockname> packages while also properly replacing them in a way the package manager would execute properly, so anyone updating wouldn't have to manually remove the i3xrocks block packages and then install the update, but I finally got that figured out and pushed it to the AUR last night.

It makes building WAY easier, it makes the PKGBUILD infinitely more readable and manageable, which makes the idea of moving to a 2.X PKGBUILD way less daunting.

Really, if I could get to a point where I knew exactly what packages from manifest.txt I could leave out and which ones I need to provide, then it should be pretty straightforward to get a PKGBUILD up and running so 2.X can go live on the AUR, and then from there we could figure out the best way to like, automate updating it. Even if it's something like a weekly script that tests the PKGBUILD to make sure it builds, and then I could pop open a VM to test the PKGBUILD in to make sure nothing's broken (like no "all the sudden ilia stopped working because of X" issues), and then it'd be just minor maintenance from there on out until some other major update.

[kgilmer]

It makes building WAY easier, it makes the PKGBUILD infinitely more readable and manageable, which makes the idea of moving to a 2.X PKGBUILD way less daunting.

Nice! If you're looking to reduce the package set further, I believe anything related to remontoire has been replaced with ilia, so that could be deprecated, along with the st customization. I have not tested st in many years. Is there something about patches that is important to your build overall?

Really, if I could get to a point where I knew exactly what packages from manifest.txt I could leave out and which ones I need to provide, then it should be pretty straightforward to get a PKGBUILD up and running so 2.X can go live on the AUR, and then from there we could figure out the best way to like, automate updating it.

It sounds like the voulage package model might fit well for this usecase. It's a simple convention in which a top-level JSON file specifies a list of things, and then an arbitrarily deep set of subdirectories each may offer a "patch" on the parent, adding/modifying/removing nodes as needed. Terminal directories become build targets which fully specify the package set required, via this process of JSON file mutation. Here's how it looks for Debian and Ubuntu in the testing stage:

~/dev/repos/voulage$ tree stage/testing/
stage/testing/
├── debian
│   ├── bookworm
│   │   └── package-model.json
│   ├── bullseye
│   │   ├── amd64
│   │   │   └── setup.sh
│   │   └── package-model.json
│   └── testing
│       └── package-model.json
├── package-model.json
└── ubuntu
    ├── focal
    │   └── package-model.json
    ├── jammy
    │   └── package-model.json
    └── lunar
        └── package-model.json

Where an element in a list is only this:

    "<package label>": {
      "source": "<git url>",
      "ref": "<git ref>"
    },

With the 3.0 beta stewing on the back burner, I'll continue work on setting up a voulage build environment running on an arch-based host. I will do this using github's self-hosted runner infrastructure. I will write a shell script that prints the following upon a Voulage build:

<package label>
<package url>
<package ref>
<target distro>
<target distro release>
<target architecture>
<build stage [ experimental | unstable | testing | release ]>

We can further refine the integration for Arch until it's well suited to the task. Are there other fields that would be helpful to print in this script @gardotd426 ?

[gardotd426]

Nice! If you're looking to reduce the package set further, I believe anything related to remontoire has been replaced with ilia, so that could be deprecated, along with the st customization.

Yeah, I've tried using voulage, but it's been a while. Really just cating manifest.txt and using cut and whatnot to print exactly what I want has been pretty easy.

Knowing I can remove remontoire when the time comes is nice. What I really would like is if we could come up with what "Regolith DE" truly entails. As in, Plasma has its packages that are required to have a usable decent experience. You have to install all of those packages, but then there are a ton of other extra KDE utilities, packages, etc that enhance the experience, but they aren't core to what Plasma is, so they're more optional. GNOME, same thing.

But with Regolith, I've had to basically editorialize what Regolith DE on Arch should be. And since it's in the AUR and cloning AUR PKGBUILDs manually and running makepkg -si will fail if there are ANY AUR dependencies included, all core packages have to be included in the meta PKGBUILD.

So, if there was like a list of a default theme, terminal, utilities, and dependencies, I could compare them with the Arch repos and see what I have to provide myself and what I can pull in as dependencies. And then I could create secondary meta-PKGBUILDs, like regolith-themes-extra for example, that people can install if they want more stuff.

But this obviously can't be a slow transition from 1.6 to 2.X, 2.X has to be all pushed live at the same time. There's way too many breakages between 1.6 and 2.X to just piecemeal update it. Right now, I created a new branch for my git repo where I am slowly just testing and screwing around with stuff, and then the 1.6 PKGBUILD is just being maintained (and being updated to try and consolodate so transitioning it to a 2.X PKGBUILD will be easier).

I have not tested st in many years. Is there something about patches that is important to your build overall?

Well yeah, st obviously requires any theming abilities to be patched in. And IIRC, either patching /etc/regolith/i3/config to change the default terminal and forcing gnome-terminal as a dependency was something I didn't want to do, or that gnome-terminal wasn't properly receiving the regolith-look stuff and showing the correct theme. I fixed the second issue, but yeah, using st was a much better option. I don't necessarily need st so if 2.X ends up working fine with another terminal, then I'll get rid of it and just add a different terminal as a dependency (even if it's gnome-terminal, since the new gnome console isn't supposed to replace gnome-terminal).

We can further refine the integration for Arch until it's well suited to the task. Are there other fields that would be helpful to print in this script @gardotd426 ?

That list is pretty great already, actually. The only thing that'd improve it off the top of my head, if it's even possible, would be if it could somehow print dependencies. Even if they're ubuntu dependencies for the time being, I could grab the .debs and see what they include and convert them to Arch dependencies. Since Ubuntu/Debian will have 4 separate packages that Arch distributes in a single package.

[kgilmer]

(WIP)

commit adding a provisional arch builder to voulage: https://github.com/regolith-linux/voulage/commit/ef447dd9377e01fa97608625dc68e61797f0848e arch build host: https://github.com/regolith-linux/voulage/settings/actions/runners/67 github workflow for arch builder: https://github.com/regolith-linux/voulage/actions/workflows/build-arch.yml Source ISO for arch build host: https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/160251/artifacts/browse/output (Arch-Linux-x86_64-basic-20230624.160251.qcow2) example build job: https://github.com/regolith-linux/voulage/actions/runs/5365917211/jobs/9735032603 example release notes: https://github.com/regolith-linux/voulage/suites/13839020208/artifacts/768457862

Primary integration point for building regolith packages for arch is this shell script: https://github.com/regolith-linux/voulage/blob/main/.github/scripts/ext-arch.sh. Each function defined here can be thought of as a call-back. We can use them all or we can only use one, depending on how arch package building workflow compares to debian.

You'll see that in each function I just log the name of the function, and in a few of them I log other data. You can see what data is printed during a build job execution, for example in this build job (https://github.com/regolith-linux/voulage/actions/runs/5365917211/jobs/9735032603) we can see some log output for xrescat build:

function called: update_changelog
function called: dist_valid
function called: stage_source
function called: build_src_package
Package Name: xrescat
Package Build Dir: /home/kgilmer/actions-runner/_work/voulage/voulage/packages
Target Repo Dir: /home/kgilmer/actions-runner/_work/voulage/voulage/repo
Manifest path: /home/kgilmer/actions-runner/_work/voulage/voulage/manifests

Distro: arch
Distro Version: testing
Target architecture: amd64

Contents of package source:
total 20
-rw-r--r-- 1 kgilmer kgilmer 1077 Jun 24 19:39 LICENSE
-rw-r--r-- 1 kgilmer kgilmer  473 Jun 24 19:39 Makefile
-rw-r--r-- 1 kgilmer kgilmer  636 Jun 24 19:39 README.md
drwxr-xr-x 1 kgilmer kgilmer  118 Jun 24 19:39 debian
-rw-r--r-- 1 kgilmer kgilmer  720 Jun 24 19:39 xrescat.1
-rw-r--r-- 1 kgilmer kgilmer 2668 Jun 24 19:39 xrescat.c

Package metadata (dependencies in deb form, etc):
Source: xrescat
Section: x11
Priority: optional
Maintainer: Ken Gilmer <kgilmer@gmail.com>
Build-Depends: debhelper (>= 11), libx11-dev
Standards-Version: 4.1.3
Homepage: https://github.com/tamirzb/xrescat

Package: xrescat
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Provides: regolith-resource-loader
Description: cat Xresources
 Simple utility which prints the value of an X resource.
function called: build_bin_package
function called: publish
CHLOG:Published arch package of xrescat in unstable arch testing amd64 (dry-run)

[gardotd426]

404

[gardotd426]

Aside from the "arch build host" link being broken, there are obviously a ton of packages that need to be removed from that build workflow, like a few of the themes, i3-snapshot, python3-i3ipc, fonts-jetbrains-mono, fonts-materialdesignicons-webfont, i3-wm, picom, i3status, and probably a bunch more, but obviously that can wait until the end (and I think it should wait until the end, like once it's up and running fine).

I'm sure you saw that the build host is missing rsync so I'm sure you'll get that fixed.

I've tried to get into github workflows once before but they're a nightmare for me, because it's like either read a bunch of documentation or have no idea what's going on, and the thing I was wanting to use them for wasn't worth the trouble, but I'm wondering, since this is an Arch build host, are you ending up with Arch packages? Like pkg.tar packages?

If so, then once that build is working, literally all we have to do is determine the packages we're leaving out, and I can create a simple PKGBUILD for the packages we are keeping, have the source array link to the built workflow pkg.tar packages, extract them and repackage them. And yeah, that's what I'd have to do, as far as I'm aware there's actually no way to have a PKGBUILD grab Arch package archives and directly install them. But that doesn't matter, they're just tar archives, and they have the correct paths already, so when they get put in the $pkgdir by the PKGBUILD, they're perfect.*

*That reminds me, we have to make sure that if you have anything that contains anything in /usr/lib, it is actually going to the right place. Arch doesn't use /usr/lib/x86_64-linux-gnu like Ubuntu does.

Though we don't have to necessarily worry about that in the buildbot. I can handle it in the PKGBUILD.

[gardotd426]

Oh I got too happy.

I read over the .yaml for the Arch build bot, and I saw it's supposedly being compiled on/for Ubuntu 20.04. Nothing using anything python or gcc is going to work on Arch if compiled for Ubuntu 20.04. At least a lot of stuff will break. Hopefully that's not a dealbreaker, like I said I have no idea how these workflows work. I'll try to make some time in the next week to read the documentation to see if I can help out, because it's not like the .yaml syntax is hard or anything.

[kgilmer]

there are obviously a ton of packages that need to be removed from that build workflow Than can be done by modifying the package model. See this commit, in which I only made changes that impact the arch build, by removing some packages from the upstream model: https://github.com/regolith-linux/voulage/commit/257962915f0856d41093ebf3f08ee3e88c9ef173

I've tried to get into github workflows once before but they're a nightmare for me,

Yes, it's a always a struggle. I believe you won't have to deal much with the github part unless you want to. The shell script functions hopefully are sufficient to perform all build tasks necessary to create and deploy packages.

but I'm wondering, since this is an Arch build host, are you ending up with Arch packages? Like pkg.tar packages?

Well that's the plan but at this point it's not generating any packages, just printing out variable etc to illustrate what's available and where.

and I can create a simple PKGBUILD for the packages we are keeping

You're welcome to add this file directly in the source repos, as we have the /debian/* stuff in there today.

I read over the .yaml for the Arch build bot, and I saw it's supposedly being compiled on/for Ubuntu 20.04.

There is only a task at the end of the build to collect some log data to produce release notes. It's entirely unrelated to the rest of the job. I've added a print statement in the script to prove we're running on an arch host :)

so this code (linked above) generates the output directly below it (listing just one package in logt data as demo):

build_src_package() {
  set -e
  echo "Build host: $(uname -a)"
  echo "function called: ${FUNCNAME[0]}"

  echo "Package Name: $PACKAGE_NAME"
  echo "Package Build Dir: $PKG_BUILD_DIR"
  echo "Target Repo Dir: $PKG_REPO_PATH"
  echo "Manifest path: $MANIFEST_PATH"
  echo ""
  echo "Distro: $DISTRO"
  echo "Distro Version: $CODENAME"
  echo "Target architecture: $ARCH"
  echo ""
  echo "Contents of package source:"
  ls -l $PKG_BUILD_DIR/$PACKAGE_NAME
  if [ -f $PKG_BUILD_DIR/$PACKAGE_NAME/debian/control ]; then 
    echo ""
    echo "Package metadata (dependencies in deb form, etc):"
    cat $PKG_BUILD_DIR/$PACKAGE_NAME/debian/control
  fi
}

function called: stage_source
Build host: Linux arch-testing-amd64-1 6.3.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 21 Jun 2023 20:46:20 +0000 x86_64 GNU/Linux
function called: build_src_package
Package Name: python3-i3ipc
Package Build Dir: /home/kgilmer/actions-runner/_work/voulage/voulage/packages
Target Repo Dir: /home/kgilmer/actions-runner/_work/voulage/voulage/repo
Manifest path: /home/kgilmer/actions-runner/_work/voulage/voulage/manifests

Distro: arch
Distro Version: testing
Target architecture: amd64

Contents of package source:
total 48
-rw-r--r-- 1 kgilmer kgilmer 6394 Jun 24 19:55 CHANGELOG.md
-rw-r--r-- 1 kgilmer kgilmer 1306 Jun 24 19:55 Dockerfile
-rw-r--r-- 1 kgilmer kgilmer 1482 Jun 24 19:55 LICENSE
-rw-r--r-- 1 kgilmer kgilmer  270 Jun 24 19:55 MANIFEST.in
-rw-r--r-- 1 kgilmer kgilmer  676 Jun 24 19:55 Makefile
-rw-r--r-- 1 kgilmer kgilmer 4242 Jun 24 19:55 README.rst
drwxr-xr-x 1 kgilmer kgilmer   98 Jun 24 19:55 debian
drwxr-xr-x 1 kgilmer kgilmer  240 Jun 24 19:55 docs
drwxr-xr-x 1 kgilmer kgilmer  768 Jun 24 19:55 examples
drwxr-xr-x 1 kgilmer kgilmer  164 Jun 24 19:55 i3ipc
-rw-r--r-- 1 kgilmer kgilmer   21 Jun 24 19:55 pytest.ini
-rw-r--r-- 1 kgilmer kgilmer  114 Jun 24 19:55 requirements.txt
-rwxr-xr-x 1 kgilmer kgilmer 2589 Jun 24 19:55 run-tests.py
-rw-r--r-- 1 kgilmer kgilmer 1487 Jun 24 19:55 setup.py
drwxr-xr-x 1 kgilmer kgilmer  370 Jun 24 19:55 test

Package metadata (dependencies in deb form, etc):
Source: python3-i3ipc
Section: x11
Priority: optional
Maintainer: Ken Gilmer <kgilmer@gmail.com>
Build-Depends: debhelper (>= 11),
               dh-python,
               python3-gi (>= 3.26.1),
               python3-dbus (>= 1.2.6),
               python3-all,
               python3-setuptools
Standards-Version: 4.4.0
Homepage: https://github.com/altdesktop/i3ipc-python
Vcs-Browser: https://github.com/altdesktop/i3ipc-python
Vcs-Git: https://github.com/altdesktop/i3ipc-python.git
X-Python3-Version: >= 3.4

Package: python3-i3ipc
Architecture: any
Depends: 
    ${shlibs:Depends}, 
    ${python3:Depends},
    ${misc:Depends}, 
    python3-xlib,
    python3-gi,
    python3-dbus
Description: An improved Python library to control i3wm and sway.
function called: build_bin_package
function called: publish
CHLOG:Published arch package of python3-i3ipc in unstable arch testing amd64 (dry-run)

[kgilmer]

Regarding the 404 link, I"m guessing it's some access thing w/ your github account and the regolith org, I'll look into it. It shouldn't matter either way though, it's ancillary. What other links or info is not working, unclear, confusing, or seems off?

I believe you should be able to make some test edit to the arch script (.github/scripts/ext-arch.sh) , merge the change back to voulage (don't worry about PR if you're only editing the arch build script), run the Arch workflow, and see your change in the build output. LMK if you're not able to do this for some reason.

[kgilmer]

Regarding getting the dependencies for a given package, I didn't look into getting deb tools in arch, although i suspect it's not a problem. So I just print the /debian/control file which does list the deps among other things to demo that it's possible to get the dependencies for a given package. Eg from the above snippet, for python3-i3ipc the real dependencies are:

    python3-xlib,
    python3-gi,
    python3-dbus

But again I suspect we can just install tools in arch to parse debian package metadata.

[gardotd426]

Yes, it's a always a struggle. I believe you won't have to deal much with the github part unless you want to. The shell script functions hopefully are sufficient to perform all build tasks necessary to create and deploy packages.

Yeah, I just want to try if I can find the time to learn some more about it so I can at least know what it can do and more on how it works under the hood so that way I know what I can suggest so I know I'm not suggesting anything patently stupid lmao.

You're welcome to add this file directly in the source repos, as we have the /debian/* stuff in there today.

Yeah, I probably will but it's probably best to wait until we actually have more of something up and running - not just the buildbot, but also the actual 2.X PKGBUILD itself, because otherwise I'll just be editing it constantly. Creating PKGBUILDs for completely unique stuff like this - this is the only DE in the AUR - always takes a bunch of trial and error, and testing, and it just takes forever. But once it's working and I want a way to better maintain it, putting a PKGBUILD in the buildbot and to try and help set up weekly auto-builds so I can be alerted if anything fails would be fantastic.

Regarding getting the dependencies for a given package, I didn't look into getting deb tools in arch, although i suspect it's not a problem

Yeah, dpkg and devscripts are both available on Arch to use for various .deb stuff. Like I said, it would just help if voulage did it because then I could create a script to grab the equivalents from pkgs.org (or anything similar) and then grab the Arch version by the pacman equivalent to apt-file. If not, I can do the same thing in a couple minutes manually anyway.

Regarding the 404 link, I"m guessing it's some access thing w/ your github account and the regolith org, I'll look into it. It shouldn't matter either way though, it's ancillary. What other links or info is not working, unclear, confusing, or seems off?

I believe you should be able to make some test edit to the arch script (.github/scripts/ext-arch.sh) , merge the change back to voulage (don't worry about PR if you're only editing the arch build script), run the Arch workflow, and see your change in the build output. LMK if you're not able to do this for some reason.

Yeah, I should be able to, there was one time a few months ago (probably longer than that now, lol) that I saw an obvious error on the regolith website and I was able to edit the repo and stuff without issue, so it shouldn't be a problem here. I'll let you know if I encounter any issues. I just opened up the build-arch.yml file and it showed the "commit changes" button which it wouldn't do if I didn't have the permissions. So that should all work fine, but like I said when the time comes I'll let you know if I run into any issues.

What's great about all this is it will make moving to 3.X way easier, it will make maintenance a breeze, and later down the road it will make it easier for either a) someone else to pick up ownership of the pkgbase if I die in a car accident (knock on wood), b) make it easier for other people to add complementary packages or create a fork that targets a different branch (like testing/experimental/etc), and most importantly c) open the door for me to optimize everything to the point where it's able to be submitted for acceptance into the actual official Arch repositories (though that's way down the line, because I'd need to get a Trusted User to agree to maintain it, or try and become a TU myself).