Generate all gap-infra tests from a single master repository

[fingolfin]

(This is somewhat related to issue gap-infra/actions#3, but strictly speaking does not require us to consolidate everything into a single repo; but I think it would be more convenient to implement and maintaining with a single repo, but only in a minor way).

Right now, making changes to the scripts driving the tests in this repository requires manually interacting with up to a dozen or repositories, applying the same change(s) to each of them, carefully. It is also easy to forget, so that an improvement or fix is only applied to a subset of all repositories that should get it.

Also, when migrating packages from -staging/-failing to -passing or vice versa, one has to update two repositories in lockstep.

Finally: when new packages are added to or removed from the package distribution, or from https://github.com/gap-packages, we may not remember to add them to the list of packages in all relevant repositories.

This all could be improved by having a single master repository which contains all the relevant data and scripts in one central place, together with a script to update the other test repositories (or, if gap-infra/actions#3 is implemented: the other branches of the master repository; to keep things simple, I'll refer to "test repositories" from here one, but obviously everything also applies if we consolidate into branches of a 1-2 central repos)

So that master repository would contain HELP.md, ci.sh, ci.g and more; plus config files for each build variant (those config files could be JSON, XML, or scripts themselves, that's a minor implementation detail).

The regeneration script could be run to regenerate all test repositories/branches in one go; additionally, it could take an optional argument(s) that allows regenerating just one (or a few) specified test repository

My vision is roughly this: the regenerate script would contain a list of branches to maintain; right now, that would probably be master, stable-4.11 and stable-4.10 (perhaps also stable-4.9?); plus one "virtual" branch release which refers to whatever is the latest actual GAP release (we could of course have this more fine grained, too, and have release-4.10.2, release-4.11.0, and even have several of them in parallel).

For each (virtual) branch, it is supposed to generate

• pkg-tests-BRANCH-passing
• pkg-tests-BRANCH-failing
• pkg-tests-BRANCH-devel
• testsuite-BRANCH

There would be one config file for each branch which lists the failing packages in it; say failing-packages-stable-4.11.json or so. Then, the list of all packages in the package distribution could be automatically determined (e.g. by downloading such a list from a central URL, where the pkg distribution mechanism puts it for us -- i.e. the same data as seen on https://www.gap-system.org/Download/upgrade.html just that perhaps it could be a separate URL for each branch, and maybe in JSON or so; say https://www.gap-system.org/packages-stable-4.11.json; in any case, I am sure we can produce such a file. And even if not, we can just revert to manually adding the list into the repository, as package-stable-4.11.json.

Using this information, it is now trivial to update both the pkg-tests-stable-4.11-staging (perhaps renamed to pkg-tests-stable-4.11-failing) and pkg-tests-stable-4.11 (perhaps renamed to pkg-tests-stable-4.11-passing) test repositories (by regenerating their .travis.yml files, and copying all other required support files, like ci.sh).

For the pkg-test...-devel variants, we'll want a list of package URLs. There are multiple ways to go about this: we could just maintain those lists in files in the master repository. Or, we could do something like what I do in https://github.com/fingolfin/gap-pkg-repos/blob/master/clone-gap-packages.py: have a fixed list of repository URLs, and then merge with the list of all packages at https://github.com/gap-packages.

(To make this really good, of course we should record for each package what the minimal GAP version it requires is. One way is to fetch the PackageInfo.g of each of these packages and look. Another would be to do this, but elsewhere, in a cron job, and again upload the result somewhere, and we just download it here. Again, these are all minor variations, in the end I don't care how exactly it is done).

Oh yeah: and once a branch drops out of support, we just remove it from the list, and (optionally) remove the helper files associated to it. Then the regeneration script will not touch the test repository/branch, and leave it in its current state forever. It is then up to us to decide whether to archive it, delete it, or whatever.

[fingolfin]

@wilfwilson this is essentially done now, isn't it? Or if not, what's left to do?

[wilfwilson]

Yeah, there's a lot here. Certainly the main topic is done, so I propose closing this. The rest of this mostly seems to concern maintaining lists of packages, which is not relevant at the moment / would depend on the package distribution.

I'll note that we've dropped the notion of passing/failing for now, but I think this is fine.