Mini-RFC: Tagging GNOME collection for updating purposes

jtojnar commented 4 months ago

Issue description

GNOME makes a release twice a year, which includes the releases of packages outside of the gnome scope (such as glib or gtk4 or more). To run GNOME updates, we currently query packages based on update script:

https://github.com/NixOS/nixpkgs/blob/45eb64faac1a6fb88d5304bd8f7e8d9e9a4dd442/maintainers/scripts/update.nix#L166-L166

But this this does not find packages that do not use update script so it is a lot of manual work. Also, for point releases, we want to separate updates targetting staging, again requiring manual filtering.

Proposal

I propose we add passthru.updateCollections attribute, which will contain a list of strings. For our use case, we would use the values "gnome" and "staging".

Additionally, we would add support for this to maintainers/scripts/update.nix.

Alternatives

Using RFC 146: meta.categories
- That is more for user-facing categories meta.tags was not approved.
- Since GNOME 47 is approaching quickly, I would go with an interim passthru.updateCollections for now and migrate to a global solution later, if one emerges.
Having a attribute list somewhere in Nixpkgs
- This would be faster to evaluate but that’s probably not an issue for updating.
- It would be easy to forget to add stuff.
Modifying maintainer teams infrastructure so that update.nix can detect lib.teams.gnome
- Would not handle staging distinction.
- Not clear how to do that in a backwards compatible way.

Not going through the RFC process since this would only affect GNOME packages and I feel like these kind of local changes are better to evolve organically.

cc @bobby285271 @getchoo @doronbehar

doronbehar commented 4 months ago

GNOME makes a release twice a year, which includes the releases of packages outside of the gnome scope (such as glib or gtk4 or more). To run GNOME updates, we currently query packages based on update script:

https://github.com/NixOS/nixpkgs/blob/45eb64faac1a6fb88d5304bd8f7e8d9e9a4dd442/maintainers/scripts/update.nix#L166-L166

So the goal is to be able to run nix-shell maintainers/scripts/update.nix (with a certain argument) upon the gnome packages that need to go through the staging -> staging-next -> master cycle, and then once that finishes to run nix-shell maintainers/scripts/update.nix again for the rest of the gnome packages?

doronbehar commented 4 months ago

I just encountered a similar problem when thinking of writing a certain automatic update related script - where I want to be able to know in advance whether an update should go through the staging branch, without running an ofborg evaluation locally (which is too resource demanding). The approaches I was thinking of are:

Count how many direct reverse dependencies exist for the package.
Lookup for previous PRs of the package and see if they targeted staging.
Parse the commit log from the last update commit up to the current HEAD, and see if there was a staging-next related merge.

jtojnar commented 4 months ago

So the goal is to be able to run nix-shell maintainers/scripts/update.nix (with a certain argument) upon the gnome packages that need to go through the staging -> staging-next -> master cycle, and then once that finishes to run nix-shell maintainers/scripts/update.nix again for the rest of the gnome packages?

Currently, for minor releases, I typically do the following:

Run update scripts for all GNOME packages.
Create two branches gnome and gnome-staging
For each of them, drop commits unsuitable for their target branch using git rebase --interactive. (Hoping I remember the where each package goes correctly.)
Open both PRs simultaneously. (Since for minor releases, lockstep update are not usually necessary except for maybe stuff like gnome-shell × mutter.)

doronbehar commented 4 months ago

I propose we add passthru.updateCollections attribute, which will contain a list of strings. For our use case, we would use the values "gnome" and "staging".

Maybe the belonging to the "gnome" collection is trivial to figure out, since the predicate you quoted already checks whether updateScript.name == "gnome-update-script". I think therefor that the only real information that would be so nice to have encoded in the expressions is whether a derivation change should go through staging or not. Hence I suggest a much simpler interface:

passthru = {
  updateScript = gnome.updateScript {
    packageName = "...";
    attrPath = "...";
  };
  updateTargetBranch = "staging";
};

It'd also be very useful for users that run their own update scripts.

jtojnar commented 4 months ago

The issue is that not all packages we consider part of gnome collection use the update script for one reason or another. This was the primary thing I wanted to address since filtering commits during rebase is not that hard, compared to gathering all the packages for update in the first place.

It might be similarly useful for e.g. Budgie, which uses nix-update-script.

doronbehar commented 3 months ago

The issue is that not all packages we consider part of gnome collection use the update script for one reason or another.

And you still want to be able to update them in the same nix-shell maintainers/scripts/update.nix command when a Gnome version is released? Because they would belong to the "gnome" update collection? You could sort of "add them to this 'collection'" by giving them an updateScript attribute with a .name == "gnome-update-script".

Another option would be indeed to add an additional attribute to the updateScript (perhaps name it collection (not plural)), and use that attribute for the update.nix predicate.

In either case I'd separate the staging/master target branch topic, from the "collection(s)" topic, since the former can be easily grasped by beginner Nixpkgs contributors without delving into the complexity of the Nixpkgs packages collections such as Gnome and Budgie, that is related to the later topic.

NixOS / nixpkgs