[Discuss] CI Jobs for macOS, msvc and msys2

lupyuen commented 3 days ago

msvc build is broken. @xiaoxiang781216 has a suggestion:

Enable One CI Job from each of macOS, msvc and msys2
Which might prevent breakage of macOS / msvc / msys2 after merging
If we don't have sufficient quota for macOS and Windows Jobs, we should reduce the Linux Arm32 Jobs

I did some analysis of the msvc breakage:

Our NuttX Dashboard shows that arm-13 is failing with the same error as msvc
So actually we should enable more Linux Arm32 Builds (instead of Windows Builds)

But I shall still enable the Windows Builds and monitor the cost. Here's my computation:

ASF has allocated a quota of 25 Full-Time Runners per week, we are using about 12 right now.
Based on the Job Runtimes: https://github.com/NuttX/nuttx/actions/runs/11630100298/usage
We estimate that msvc + msys2 will cost us 60 mins of GitHub Runner Minutes (Windows Runners cost twice as much as Linux Runners)
So we might be OK to enable msvc + msys2 Builds
But I will still shut them down if they overrun our quota

Sorry I can't enable macOS Builds right now:

macOS Runners cost 10 times as much as Linux Runners. To enable One macOS Job, we need to disable 10 Linux Jobs! Which is not feasible.
Our macOS Jobs are in a bad state right now, showing too many warnings. We need someone familiar with Intel Macs to clean up the macOS Jobs. https://github.com/NuttX/nuttx/actions/runs/11630100298/job/32388421934 https://github.com/NuttX/nuttx/actions/runs/11630100298/job/32388422211

But can we still prevent breakage of Linux / macOS / msvc / msys2 Builds?

Nope this is simply impossible. In the good old days: We were using far too many GitHub Runners. This is not sustainable, we don't have the budget to run all the CI Checks we used to.
So we should expect some breakage to happen. We have to be prepared to backtrack and figure out which PR broke the build.
That's why we have tools like the NuttX Dashboard, to detect breakage earlier without depending on GitHub CI.
Also we should show some love and respect to NuttX Devs: Previously they waited 2.5 hours for All CI Checks. Now they wait at most 1.5 hours, I think we should stick to this.

No response

lupyuen commented 3 days ago

NuttX Dashboard shows these broken builds, enabling the arm-08, arm-10, arm-13 builds will help:

xiaoxiang781216 commented 3 days ago

@lupyuen the plan is good to enable msvc and msys2, but disable macOS since macOS consume a lot of quota.

lupyuen commented 3 days ago

Yep fully agree, macOS Builds will only run in our Mirror Repo. So they won't be charged to NuttX. Thanks!

apache / nuttx