Github CI is proprietary and limiting

[gdt]

It would be nice for unison to support other CI systems, specfically ones that

• are actually open source
• support more than the usual suspects of operating systems
• support more than the usual i386/x86_64 CPU architectures

Inspired by #443, which is an example of this problem, but too narrow to be carried in the bugtracker.

[mvglasow]

I have some experience with Gitlab CI, where I have set something up for my projects (mostly Android, some Java on Linux). It basically supports anything for which Docker images are available, though I don’t know if that is the case for the OSes Unison intends to target. AFAIK the CI component is also part of the open-source Community Edition.

If we opt for that, however, it would make sense to either set up a Gitlab mirror which gets updates automatically (either via push or pull, afaik that can be set up), or move to Gitlab altogether (they support moving Github projects there).

As for third-party CIs, I know the guys over at Navit are on CircleCI (with a Github repo) and are currently working on migrating towards Github CI, as third-party CI integration has caused us various headaches.

[gdt]

While I'm sympathetic to gitlab vs github, that's probably a bridge too far this minute.

But if someone wants to set up a mirror, and have some additonal CI, that sounds great. (I am mostly triaging tickets, overseeing hitting merge on proposed changes, tagging releases, etc., and thus have no unison cycles to actually do anything, but many hands make light work.)

[gdt]

Stunningly, github self-hosted runners are also limited to only 3 operating systems, macOS and Windows x86_64 only, and for GNU/Linux only, also arm32 and arm64.

[gdt]

CI for all Ubuntu 18.04 are failing to start. It remains to be seen if this is a glitch or if that's been withdrawn.

[gdt]

Information on future changes is hard to discover, so note-to-self:

https://github.com/actions/runner-images/issues/5583 https://github.com/actions/runner-images/issues/6002 https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/ https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

[mvglasow]

As I am currently messing with GHA for this project, I notice you guys are making quite aggressive use of GHA features. Migration would definitely get easier if you could start minimizing dependencies on GHA features, to make the CI workflow as portable as possible. Probably the very first thing to address is uses, i.e. referencing Github Actions outside this repo, which is currently being used for a few things, most importantly to install OCaml. For best portability, the CI workflow should be a self-contained collection of jobs, steps and dependencies, with as much logic as possible in shell scripts.

[gdt]

A fair point and thank you for making it. It's been a long journey to where we were, and surely a long journey forward, within a changing world. I'm very sympathetic to the idea of making the CI process more portable. Right now we're in a pruning phase, but maybe we're done pruning, and even then it's size of runs, not complexity.

I feel underclued about the details, but if you are inclined to make a PR that perhaps takes a small step towards reducing reliance on GHA details while being neutral about what the CI build does, or something kind of like that, I will definitely pay attention.

[tleedjarv]

I notice you guys are making quite aggressive use of GHA features.

Looks can be deceiving. Yes, the workflow file is enormously complex and I don't like it either. But it's nothing inherent to the project's needs. It's just for GHA and is not coming with should the CI be run elsewhere. We avoid GHA dependency as much as possible. Everything you see there is boilerplate and the result of how GHA works. It's not "the CI workflow", it's "the GHA workflow".

The workflow is not an issue when it comes to migration. Platform availability is.

[mvglasow]

Having tried out a few things on both Github and Gitlab (including reproducing parts of the Unison CI flow on Gitlab), here are some of my experiences:

For one, I would suggest moving away from Github’s CI images to standard Docker images with some of our own customizations, which we could then publish as separate Docker images. I have done this for other projects on Gitlab in the past. GHA also allows running on arbitrary Docker images, and as I understand, Docker images can also be maintained on Github. Bottom line: we could tackle this before moving away and move this setup to a future new “home” with relatively little effort. As a side effect, this would presumably speed up our CI jobs, in addition to making them easier to read (as all setup-related stuff would be moved into the Dockerfile). In the process, we could replace references to other GHAs (currently OCaml setup is the one we really need) with something more portable and would make the whole CI setup more portable.

As for Gitlab, there is no support for steps in jobs – every job is essentially one bash script. However, that should be less of an issue if we can move all of the setup stuff in a Dockerfile rather than running it again for each job. If structuring is needed, we could split jobs into multiple dependent jobs.

A minor one is that on Gitlab, CI runs as root on the image, whereas GHAs run as a non-privileged user. This should not really be an issue; for the most part, dropping the sudo before privileged commands is all we need to do.

Platform availability: Linux is the default; Windows and MacOS are currently in beta.

I haven’t yet found a way to set a custom shell in GitLab. Linux jobs run in bash; not sure about other platforms. For building and testing, this should be sufficient; if setup takes place in a Dockerfile (which would be platform-specific anyway), this should not be an issue either.

Some actions happen automatically in GitLab (whereas in GHA they must be specified explicitly): checking out code, uploading artifacts (paths are specified as job parameters and artifacts are collected at the end) as well as retrieving/updating caches (cache paths and keys are specified as job parameters; parameters in keys can be used to control how caches are shared between pipelines, or jobs in a pipeline).

Artifacts are collected at the end of each job if a path is configured, and can be downloaded from the pipeline. Not sure if there is an action to “publish” an artifact – we”d have to look into that.

Billing: My personal Gitlab account (free account created in 2014) comes with 2000 free CI minutes per month; an organization I created in 2018/2019 gets 400 CI minutes per month. These may vary depending on when the account or organization was set up. One CI minute equals one job running for a minute (two jobs running in parallel will consume two CI minutes per minute). Registering a project as a FOSS project will effectively double your quota (CI minutes used will be counted with a factor of 0.5 towards your remaining quota); forks of a recognized FOSS project get even bigger discounts. During the beta phase, a CI minute is a CI minute, regardless of the platform, but this may change at the end of the beta phase, where Windows and Mac may consume more than one CI minute per minute of runtime. Additional CI minutes can be purchased.

[tleedjarv]

Thank you for sharing your experiences.

For one, I would suggest moving away from Github’s CI images to standard Docker images with some of our own customizations, which we could then publish as separate Docker images. I have done this for other projects on Gitlab in the past. GHA also allows running on arbitrary Docker images, and as I understand, Docker images can also be maintained on Github. Bottom line: we could tackle this before moving away and move this setup to a future new “home” with relatively little effort. As a side effect, this would presumably speed up our CI jobs, in addition to making them easier to read (as all setup-related stuff would be moved into the Dockerfile). In the process, we could replace references to other GHAs (currently OCaml setup is the one we really need) with something more portable and would make the whole CI setup more portable.

I very much agree with this. GHA workflows are much too complex even for the simplest of tasks. And our needs are actually very simple. The only difficult need is wide platform coverage. Non-portability makes this complexity so much worse.

As for Gitlab, there is no support for steps in jobs – every job is essentially one bash script. However, that should be less of an issue if we can move all of the setup stuff in a Dockerfile rather than running it again for each job. If structuring is needed, we could split jobs into multiple dependent jobs.

Good riddance! It's the steps that are making even the simplest of GHA workflows so very complex.

Platform availability: Linux is the default; Windows and MacOS are currently in beta.

Lack of available platforms is one of my biggest concerns with Docker. While GHA does not natively support other platforms either, there are workflows that leverage VirtualBox installed on a macOS host which makes it possible to run CI scripts on BSDs and Solaris, for example. (And yes, I've made Unison CI run successfully on those.)

I haven’t yet found a way to set a custom shell in GitLab. Linux jobs run in bash; not sure about other platforms. For building and testing, this should be sufficient; if setup takes place in a Dockerfile (which would be platform-specific anyway), this should not be an issue either.

I don't think this is a problem at all. The actual CI script does not have to be interpreted by the initial/only shell. The script can be made very platform-independent. It doesn't even need to be written in any shell or other interpreted language. The need for different shells in GHA worfklow is caused by GHA complexity and the aforementioned "steps" (and also by the expectation that OCaml projects need a POSIX build environment, but that can and will change).

The CI "script" in its simplest form does not require any shell capabilities at all, beyond just executing a batch of commands:

## Build OCaml compiler (even better if it could be installed by system package manager)
wget https://github.com/ocaml/ocaml/archive/refs/tags/4.14.1.tar.gz
tar zxf 4.14.1.tar.gz
cd ocaml-4.14.1
./configure
gmake
gmake install
cd ..
rm -rf ocaml-4.14.1

## Build unison
gmake

## Run unison tests
# Built-in tests
gmake test
# RPC tests over local socket
mkdir localsocket
chmod 700 localsocket
# Separate backup dir must be set for server instance so that the central
# backup location of both instances doesn't overlap
UNISONBACKUPDIR=./src/testbak4 ./src/unison -socket ./localsocket/test.sock &
sleep 1 # Wait for the server to be fully started
test -S ./localsocket/test.sock
./src/unison -ui text -selftest testr3 socket://{./localsocket/test.sock}/testr4 -killserver

(the only platform-specific thing here is the Unix domain socket and that is not a requirement; it could be replaced with a TCP socket or even ssh)