[FEATURE] Revising the Arm32 Targets for CI Build

lupyuen commented 1 month ago

Is your feature request related to a problem? Please describe.

Over the weekend we had CI Build Issues that took a while to resolve:

Our CI Build for Arm32 Targets requires 2.5 hours to run (arm-01 to arm-14). It takes some time to catch all the errors, patch them, re-run the build. And we might hit errors again. Let's improve this.

Describe the solution you'd like

What if we could validate the defconfig files earlier, before any builds? (Suggested by @GUIDINGLI)

Might be possible, I'll check through the CI Build Script (Done! See the PR below)

Do we really need to build so many Arm32 Targets every time?

If we build only the latest targets (e.g. SAMV7, Goldfish), then the problems will show up earlier. (And reduce our cost of GitHub Actions)
Or we should move SAMV7, Goldfish etc to the first job arm-01, keep the job small, so that it will fail earlier.

These are the Arm32 Targets that we should keep for the CI Build, suggested by @GUIDINGLI:

armv8-m, armv8-a, armv8-r
armv7-m, armv7-a, armv7-r
armv6-m
For each of them: Keeping two/three boards is enough
For each of them: Build for Flat mode, Kernel mode (if supported), Qemu (if supported)

Here's the complete list of 932 Arm32 Targets (search for arm-01 to arm-13)

Hi @xiaoxiang781216 @acassis @cederom what do you think? Thanks!

Describe alternatives you've considered

No response

Verification

[X] I have verified before submitting the report.

GUIDINGLI commented 1 month ago

@pkarashchenko @masayuki2009

Let's build & run the board which really needed: https://github.com/apache/nuttx/pull/14007

cederom commented 1 month ago

Yes in the light of overloading CI we need optimizations. Also current complexity makes things broken quite often and hard to analyze.
Optimizations should not jeopardize quality of testing - so we still should test all possible configurations (as mentioned above) but maybe on lower count of boards or with smaller tasks that are easier to manage I agree.
We still need to build on Linux, macOS, Windows. There are mostly used, others are not available on GitHub (i.e. BSD).
I would suggest to add details to worker names on what is built inside (i.e. linux-arm-01-armv8m-sim-stm32, linux-xtensa-01-esp32, linux-rv-01-esp32c3, etc). Maybe its more convenient to use single worker per OS like in msys2 right now there is only one worker that makes all of the tasks (decreased setup time per build)? Some top-bottom architecture would be nice to have here :-)
We should really think more about distributed testing platforms that would involve various boards that people have at hand. This would take burden from CI and would reveal other problems not caught by CI. But its a separate complex project I know. And with underlying tools that does not provide self-compatibility (i.e. recent problems with Python and PIP changes) its going to be maintenance nightmare. Welcome to the brave new world of constant change ;-)
GH CI is amazignly powerful hardware, we may also consider SIM to perform some initial in-RTOS testing (like ostest) to make sure build works as expected.

lupyuen commented 1 month ago

Hi @GUIDINGLI: This script will validate all defconfig files for all targets: arm-01 to arm-14, risc-v-01 to risc-v-06, etc. Lemme figure out how to add this to our CI Workflow, meanwhile we can run it manually. Thanks!

https://github.com/lupyuen/nuttx-riscv64/blob/main/check-config.sh

Update: Here's the PR that will validate all defconfig files before building: https://github.com/apache/nuttx/pull/14317. Now waiting for the CI Backlog to clear. Thanks!

Update 2: Nope the PR doesn't work sigh

Update 3: The PR works OK now yay!

apache / nuttx