Alternative strategy for Improved CI Build and Deployment times

[DMills27]

Chore

Describe the chore

The current setup uses Github Actions to build and deploy from this repository. However, it suffers from issues like needing to rerun the entire pipeline to debug or test changes, leading to unnecessary wait times instead of caching unchanged results. Additionally, the pipelines themselves aren't necessarily the most optimised since they run on vCPUs.

I suggest Garnix as an alternative for (certain aspects of) the CI process. Garnix is a Nix based CI build tool. In order to use it one simply needs to have a flake.nix file in the repository under consideration. In making this suggestion I'm not arguing for overhauling the existing architecture but simply offering an alternative to the current build process that will work in a more performant manner while not intruding on what already exists, aimed mainly at those who wish to self-host. This comes with a number of benefits, such as:

• Blazingly fast build times through higher levels of parallelism as well as making use of CPU shares, rather than vCPUs.
• Sharing the same cache among multiple targeted repositories.
• Making it much easier to deploy sophisticated integration and E2E NixOS tests akin to this one which starts a primary MySQL instance and synchronises two secondary instances. It then simulates downtime by stopping one secondary instance and adding data to the primary, then restarts and re-synchronises the secondary instance. Tests like these are useful for testing the robustness of database migrations.
• Allowing one to seamlessly debug failed builds since Garnix provisions a server upon each build, which one can SSH into and figure out what went wrong. Disclaimer: This is a paid feature though.

Both of these options can exist alongside one another and provide points of comparison and contrast. Moreover, they can supplement one another as well since Garnix supports integration with pre-existing Github actions by specifying it in the garnix.yaml file of the repository which makes it possible to offlet some of the work done by Github Action to Garnix to reduce overall time spent on different aspects of the build process.

I forked this repository and to see how long it would take for Garnix to build and it took around ~20 minutes for the flake alone. I didn't try to integrate the other Github actions because it was onerous to do so since some of them needed me to set-up secrets, etc.

The second time it took even shorter since it remembered the packages that were built from a previous commit, and produced the build in a fraction of the time it took on the initial run.

[olirice]

thanks for opening

close with explanation in https://github.com/supabase/postgres/pull/1079#issuecomment-2261355276

but feel free to follow up if you'd like to discuss further