Build proposal: Add continuous integration to PRs, and have tests run automatically.

[blaisemccourtney]

@sprunk suggested in Discord as I understood him that something like continuous integration could be a useful addition to the engine repository, so I am making an issue on it. I did not find any existing issues in the repository about this topic. I do not know how easy or difficult this might be to set up. It might also require build servers. However, Beyond All Reason is open source, and some companies might offer services for open source projects for continuous integration for free.

[p2004a]

According to what I remember from conversions about it on Discord:

There is no GitHub Actions CI in engine at the moment because engine build takes like 40m-1h and folks didn't think it's reasonable to wait that long. They also didn't want to spend all the GitHub free minutes. Also there is assumption that open PRs are build and tested locally by people. And there are ~no tests to run, and no consistent code style to check, so 🤷‍♂️ even less benefits.

I personally think it still would be good to add, so if any main engine devs changed their minds, I will add it to my backlog.

[p2004a]

Re build time: we could look into using other runners that allows spec customization, not standard ones like:

• https://namespace.so/
• https://buildjet.com/for-github-actions
• ...

[lhog]

Yeah, that would be truly great to have however as stated we hit two important resource constraints: 1) Amount of github free minutes 2) Amount of disk space available for caching. Right now building of a new branch evicts the cache for the previous branch so switching branches always results in ~1h of build time, which directly contributes to the bullet number one.

[Jordan-Cottle]

Would it be worth setting up a self hosted runner so that build minutes aren't an issue?

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners

There's probably a cloud free-tier out there somewhere that isn't being used by the BAR team already that would make a decent runner

[p2004a]

It's not that easy as by default the are not designed for public repositories https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#self-hosted-runner-security and it's not theoretical https://johnstawinski.com/2024/01/11/playing-with-fire-how-we-executed-a-critical-supply-chain-attack-on-pytorch/

Maybe somebody already wrote some soft to deal with that. But still, machine would be basically cold and not doing anything almost all the time, and would want a large machine for quicker builds: not very suitable from any free tiers. Optimally, we need a bunch of compute, like 8-16 (maybe even 32, not sure if build would scale well on more) cores for relatively short time.

That's why I would lean towards some hosted pay-as-you-go solution: needs research into which one, what would be cost be, do they offer features we need (caches size etc) and so on.

[Jordan-Cottle]

Dang, I'd read about the issues a while back. I had figured github would have some kind of way to configure things safely, but it looks like the situation hasn't changed much.

I use gitlab mostly instead of github for things, and the ci/cd over there is a lot nicer for setups like this it seems.