The current solution to run a VM on MacOS shared runners using Vagrant is becoming very unreliable and almost always breaks. Replace it with Testing Farm [1], utilizing the "Schedule tests on Testing Farm" GH Action [2].
Advantages:
more reliable
allows us to test also on aarch64
currently no usage limits for public projects (will likely change)
Disadvantages:
requires an API key to be stored in the project's secrets (under the name TESTING_FARM_API_TOKEN)
GitHub won't allow the target project's secrets to be used when running a workflow on a pull request from a fork, therefore PRs won't be automatically tested unless the contributor sets their own API key in their fork
only people with a Fedora account in the fedora-contributor group can currently obtain an API key on their own (others would need to ask for it via email)
no real-time view of test progress (may become available in the future via the artifacts view)
The new CI runs the testsuite on a similar testing matrix as the old one, although it only tests on the latest Fedora version and additionally tests on the aarch64 architecture. It also runs the NFS tests (./tools/nfs.sh), which the old one didn't.
The current solution to run a VM on MacOS shared runners using Vagrant is becoming very unreliable and almost always breaks. Replace it with Testing Farm [1], utilizing the "Schedule tests on Testing Farm" GH Action [2].
Advantages:
Disadvantages:
The new CI runs the testsuite on a similar testing matrix as the old one, although it only tests on the latest Fedora version and additionally tests on the aarch64 architecture. It also runs the NFS tests (./tools/nfs.sh), which the old one didn't.
[1] https://testing-farm.io/ [2] https://github.com/marketplace/actions/schedule-tests-on-testing-farm