Automated PKGBUILD updates

[mrzor]

Hello there! Thanks to your hard work packaging this, I'm in the process of upgrading all my workstations to Arch, once more.

I had great success - but ran into some deprecated packages a few days ago which prompted me to start an automated upgrade script to update PKGBUILD from upstream.

I noticed you pushed several updates today, I imagine the PKGBUILD is back in order, thanks for that.

You can find the script at https://github.com/mrzor/regolith-de/blob/automated-updates/pull-upstream.py - it's not quite ready to be PR'd - no docs, no CLI, and most importantly, it doesn't update SHA256s yet. I haven't found where to source them from upstream, hopefully I'll figure this one out sometime next week.

You can have a look at the preliminary result at https://gist.github.com/mrzor/e06faeb76f005d485273fce5a37c211b

I thought I'd drop by to say thanks and let you know this is in progress. If you'd like the script to go one direction or another, I'm open to suggestions!

[gardotd426]

Yeah an update script would be really nice. Also, there are some people talking to the Regolith creator about getting away from Debian-based packaging, to make it so we can just build the stuff from source, but that will likely be a while (though he is very open to the idea).

I know jack shit about Python, but I'll take a look at it.

And yeah, downloading the .deb packages and sha256ing them is how I've been getting the sums, not sure if there's an automatic way to figure it out. I'll look into it.

[gardotd426]

Oh and we're tracking release, not stable.

[mrzor]

Thanks for answering!

The script can track release, it's just one edit. I'll make release the default before I PR. If other persons would rather track another channel, they will be able to run the script themselves to do so.

I'd personally rather track stable - upstream is very conservative w.r.t. release. Stable feels more Arch-ish. I'm happy to keep this as a personal thing - not trying to make you change your mind on this.

I asked Ken on #regolith-users about hashes, and it doesn't look like he signs or hashes the debfiles himself. If there's anything done on that front, it has to come from launchpad. I haven't found anything yet though.

W.r.t. distribution-agnosticism from Regolith maintainers, Ken does indeed sound enthusiastic about it. Perhaps we can contribute on that front once it gets rolling. It's certainly a much bigger endeavor than just having a set of Arch-compatible Regolith packages.

Meanwhile, I'll finish up that script. The easiest way is to force download of the relevant archives (by the script) and run sha256sum on them. I'll keep this sequential and slow for simplicity's sake. Watch this space!

[gardotd426]

I'd personally rather track stable - upstream is very conservative w.r.t. release. Stable feels more Arch-ish. I'm happy to keep this as a personal thing - not trying to make you change your mind on this.

Well, after watching for about 9 months (hard to believe that I started this 9 months ago, but time is meaningless at this point so I guess it makes sense), stable is nowhere near as updated (or complete) as release. Also, the individual packages are updated piecemeal on Stable, whereas the release branch is more updated (mostly) all at once, which makes way more sense when it comes to practicality. If all that changes I'm not opposed to moving to stable (or even unstable), but for right now release makes the most sense (I actually use to track stable in the first iteration, but it wasn't working out).

[mrzor]

PR #21 has been opened with the result of one extra afternoon of work. It now downloads everything and computes sha256sum. I've added some instructions to help people setup the python deps.

I think there's an extra bug lurking somewhere, but this should work well enough for you to make PKGBUILD updates whenever you want.

Thanks to the script I was able to ./pull-upstream -c stable -f then makepkg -f then sudo pacman -U *.pkg.tar and update the whole stack in a very smooth way.

This script was written with an implicit requirement of making your life easier and not disrupt the way you do your thing too much. Because of this, it edits the PKGBUILD instead of generating one. This is a bit error prone, and I don't claim it will support any other PKGBUILD variations that could be introduced.

Generating the full PKGBUILD from a template would be more future-proof, but it will be slightly more complex for yourself or other contributors. I can contribute such a thing if you see its use.

[mrzor]

Scripts warns about i3xrocks-info_3.5.6-1_amd64.deb because while it's in the sources, it doesn't appear to be installed. I suppose that's something you'd like to have a look at :)

[gardotd426]

Well, I like it a lot, but there's a couple things...

For one, it's changing the URLs from my predefined "${url1}", "${url2}", etc. So now I get a source array that looks like this:

source=("${url2}"/ayu-theme_0.2.0-1ubuntu1~ppa1_amd64.deb
        "${url2}"/cahuella_1.0.2-1_amd64.deb
        "${url2}"/i3-snapshot_1.0-1ubuntu1~ppa1_amd64.deb
        "${url2}"/i3xrocks_1.3.4-1_amd64.deb
        "${url2}"/i3xrocks-battery_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-cpu-usage_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-focused-window-name_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-info_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-key-indicator_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-media-player_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-memory_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-net-traffic_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-nm-vpn_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-openvpn_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-temp_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-time_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-volume_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-weather_3.5.6-1_amd64.deb
        "${url2}"/i3xrocks-wifi_3.5.6-1_amd64.deb
        "${url2}"/nordic_1.6.5-1ubuntu1ppa1_all.deb
        "${url2}"/paper-icon-theme_1.5.723-201905252133~daily~ubuntu19.04.1_all.deb
        "${url2}"/plano-theme_3.36-1-1regolith1_all.deb
        "${url2}"/plymouth-theme-regolith_1.0.3-1_all.deb
        "${url2}"/regolith-compositor-picom-glx_1.1.3-1_amd64.deb
        "${url2}"/regolith-default-settings_1.0.5-1groovy_amd64.deb
        http://ppa.launchpad.net/regolith-linux/release/ubuntu/pool/main/r/regolith-desktop/regolith-desktop-complete_2.108-1bionic_amd64.deb
        http://ppa.launchpad.net/regolith-linux/release/ubuntu/pool/main/r/regolith-ftue/regolith-ftue_1.1.1-1_amd64.deb
        "${url2}"/regolith-gdm3-theme_2.0.0-1ubuntu1~ppa1_amd64.deb
        "${url2}"/regolith-gnome-flashback_2.6.2-1_amd64.deb
        "${url2}"/regolith-i3-gaps-config_2.8.4-1_amd64.deb
        "${url2}"/regolith-i3xrocks-config_3.5.6-1_amd64.deb
        "${url2}"/regolith-lightdm-config_1.0.6-1_amd64.deb
        "${url2}"/regolith-look-ayu_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-ayu-dark_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-ayu-mirage_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-cahuella_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-dracula_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-gruvbox_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-lascaille_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-nord_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-pop-os_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-solarized-dark_2.6.14-1_amd64.deb
        "${url2}"/regolith-look-ubuntu_2.6.14-1_amd64.deb
        "${url2}"/regolith-rofication_1.3.1-1_amd64.deb
        "${url2}"/regolith-rofi-config_1.3.1-1_amd64.deb
        "${url2}"/regolith-st_0.8.2-1ubuntu20ppa5_amd64.deb
        "${url2}"/regolith-styles_2.6.14-1_amd64.deb
        http://ppa.launchpad.net/regolith-linux/release/ubuntu/pool/main/r/regolith-system/regolith-system_1.5.0-1_amd64.deb
        "${url2}"/solarc-theme_800c997-1_amd64.deb
        "${url2}"/ubiquity-slideshow-regolith_138.5-ubuntu1~regolith1_all.deb
        "${url2}"/xrescat_1.2.1-1_amd64.deb
        flashback.patch
)

You can tell which ones were monkeyed up by the script, which are the ones that received updates.

Second, (and second most important), it's pulling in bionic versions of packages. We always want to use the latest release, which would be groovy rather than bionic.

Third (and most important of all), it's breaking the PKGBUILD.

These three .debs were updated in my PKGBUILD.new after running the script:

regolith-system_1.5.0-1_amd64.deb regolith-ftue_1.1.1-1_amd64.deb regolith-desktop-complete_2.108-1bionic_amd64.deb

And now look at the build part of the PKGBUILD:

extract_deb "${srcdir}"/regolith-compositor-picom-glx_1.1.3-1_amd64.deb
    extract_deb "${srcdir}"/regolith-default-settings_1.0.5-1groovy_amd64.deb
    extract_deb "${srcdir}"regolith-desktop-complete_2.108-1bionic_amd64.deb
    extract_deb "${srcdir}"regolith-ftue_1.1.1-1_amd64.deb
    extract_deb "${srcdir}"/regolith-lightdm-config_1.0.6-1_amd64.deb
    extract_deb "${srcdir}"/regolith-i3-gaps-config_2.8.4-1_amd64.deb
    extract_deb "${srcdir}"/regolith-rofi-config_1.3.1-1_amd64.deb
    extract_deb "${srcdir}"regolith-system_1.5.0-1_amd64.deb
    extract_deb "${srcdir}"/ubiquity-slideshow-regolith_138.5-ubuntu1~regolith1_all.deb
    extract_deb "${srcdir}"/regolith-i3xrocks-config_3.5.6-1_amd64.deb
    extract_deb "${srcdir}"/regolith-rofication_1.3.1-1_amd64.deb

Do you see what's wrong?

Hint: ar: /home/matt/nvme2/dev/regolith-auto/srcregolith-desktop-complete_2.108-1bionic_amd64.deb: No such file or directory

There's no / between `"${srcdir}" and the package name. There has to be a slash. So whatever part of the script you have that replaces the package names in the build array, you gotta not remove the slash.

would it not be possible to just sed version numbers? package NAMES obviously aren't going to change, and if they do your script wouldn't handle the name change anyway (right?), so could you not just replace the version number?

But yeah, if you can get it actually working I'll be thrilled, because it's so much easier than me having to do all this shit manually like I have been, I'm already setting up github-cli to try and automate updates twice a week or something in anticipation lol.

[mrzor]

Regarding number 1, I couldn't work out something nice ; the URLs you were using previously are not exactly the ones I could scrape. On top of that, dealing with bash variable expansions was lots of complexity (what if url2 changes tomorrow ...), so I worked around it. It's much easier for the script to have full size URLs in the PKGBUILD.

Regarding number 2, I'm quite surprised. http://ppa.launchpad.net/regolith-linux/release/ubuntu/pool/main/r/regolith-desktop/ Looking there we notice that the groovy version is 2.106 while the bionic one is 2.108. Right now, the way it works is that the script picks up the ... last one it sees. It's completely stupid. It didn't occur to me that some packages would have several versions side by side in the repo - the ones I checked were not like this.

Regarding number 3, that's a breaking bug I must have introduced in the course of fixing other bugs :facepalm: Will fix.

[mrzor]

Regarding 2, I reported this on Slack - I don't believe that generally latest Ubuntu release means more recent, especially when package versions says otherwise. I think we should trust the version number more in that case.

[mrzor]

Point 3 is fixed in latest pushed version.

Wrt. point 2, I'd rather see upstream fix the repo contents rather than implement an ubuntu version parser and a custom version comparison (even though 80% of it is commented out in the code right now).

I don't intend to address point 1 in code at this point, unless you believe it to be a big issue.

would it not be possible to just sed version numbers? package NAMES obviously aren't going to change, and if they do your script wouldn't handle the name change anyway (right?), so could you not just replace the version number?

You are right - name changes are not taken into consideration.

It replaces based on {name}_{version}. Imagine having a_1.0 and b_1.0 but only b gets bumped to 1.1 - replacing all 1.0 into 1.1 wouldn't work.

[gardotd426]

So I just realized, if I'm going to put this in the AUR, this script wouldn't be a part of it, as that kind of screws up what a PKGBUILD is supposed to be. If I distributed your script with the PKGBUILD in the AUR, and an updated package broke something, I wouldn't know, and the user would have to find out on their own, and that's not something I really want to put out there. It makes way more sense for me to create a second "regolith-dev" or "regolith-scripts" repo, for things that aren't part of the makepkg process (because other stuff shouldn't be included), and put this script in there, and then run it weekly or something. That way I can test out updates before pushing to the AUR.

Does that sound good to you?

[mrzor]

Anything you prefer :)

I wrote this to help you update the PKGBUILD once I realised how painful it was - and to be frank, it also scratches an itch of mine cause I wanted to use unstable.

If you find it clearer or helpful to have several repos, I have no objections.

for things that aren't part of the makepkg process (because other stuff shouldn't be included)

I'm not sure about that "shouldn't". I would argue in favor of convenience and pragmatism. We do this on our limited free time, if keeping it simple means breaking a soft rule, why not.

[mrzor]

And to answer your first comment : no the script is not a PKGBUILD or what a PKGBUILD should be (or ... could it ... why not ... would probably raise many eyebrows ... :thinking: ). It's meant to help you update the current PKGBUILD.

Notably, it doesn't concern itself with the way you editorialized regolith into a smaller package set, and it doesn't try to pull in more than what you chose from upstream. In that aspect, the original PKGBUILD is still very much your valuable, original work.

[gardotd426]

Hey let me know if you figure out anything about what I mentioned to you the other day. I'll close this issue for now, just comment on the merge request.

Oh and the package is live in the AUR. regolith-full is the meta-package, regolith-de is the pkgbase containing the 5 packages that make up the DE.

So if you're using an AUR helper and want the full DE, you can just run yay -S regolith-full, or paru -S regolith-full, or whatever AUR helper you like. If you're cloning the AUR repo and building manually with makepkg, obviously you wouldn't use the meta-package, you would just git clone https://aur.archlinux.org/regolith-de.git, cd into regolith-de, and run makepkg -si.

[mrzor]

Hey there. I'll just go along with what you think is best. No objections to putting this in a regolith-scripts repo or what have you.