ci: Track build time for Hello World multi-platform

[kartben]

Is your enhancement proposal related to a problem? Please describe.

The newly introduced Hello World Multiplatform build is a great way to track that we have a working out-of-the-box experience for everyone, in particular people checking out Zephyr for the first time. It would be nice to go one step further and capture the evolution of the "time to Hello world" over time.

Describe the solution you'd like

Capture time to build hello world on each platform. Would be nice to not only track the full "zero to hello world" time, but also "time to install sdk", "time to perform the first west update", etc. so that we can pinpoint potential regressions better (ex. a HAL growing substantially, etc.) The captured metrics (csv? json? whatever...) should be pushed to our dashboarding infrastructure so that everyone can easily visualize trends and flag any regression.

Describe alternatives you've considered

None

Additional context

If we think this is a good idea, it might be a challenge to make sure captured times remain meaningful over a long period of time, i.e. should the specs of the runners used change, this will of course impact build time.

Also good to note that the reported time will likely be less than the one experience by a real-world user on a similar machine (i.e. with clone with fetch-depth=0), but what matters is the trend over time, not only the absolute numbers.

cc @fabiobaltieri (may I assign this to you? ;-))

[stephanosio]

I do not see how this can work in practice.

Build time is highly dependent on the host CPU performance.

Even for the same type of instance on AWS, the jobs can be scheduled onto many different types of CPUs (I've seen at least 4 so far) with varying clock frequencies and cache sizes. To make matters worse, AWS vCPUs are "threads" and not "cores," meaning the build time will significantly increase if a CI job ends up running on more than one thread on a host.

And with the upcoming zephyr-runner v2, jobs can even be scheduled onto multiple generations of x86 CPUs (Sandy Bridge, Ivy Bridge, Haswell, Broadwell, Zen 2, Zen 3) and even multiple architectures (x86 and AArch64).

I do not think it makes sense to track build time.

[kartben]

Thanks, @stephanosio! Completely agreed that build time can fluctuate quite a bit and pointed it out in my initial comment. I guess I was less concerned about this due to Hello World using the GH hosted runners, that I would (wrongly?) wrongly assume to be more "stable" in terms of specs.

Looking at several months of nightly builds of an out-of-tree project of mine using Fabio's action is showing pretty steady numbers (+/- 2%) between each run, so even even if tracking only works for months at times and get an occasional shift should GH change the specs of their runners, I still think it's valuable data to surface.

[nordicjm]

To add to the above, we know building on windows is just a mistake in general because of the huge overhead that the antivirus adds and that is completely out of the users control, it's possible that the AV might be updated by microsoft in the future and made faster or slower, and it would wrongly skew numbers

[stephanosio]

build time can fluctuate quite a bit

"quite a bit" is more like by +/-100%.

using the GH hosted runners

GH hosted runners are no different from AWS. They are likely one or more types of VM instances from Azure, which can consist of multiple types or even generations of CPUs with all varying clock frequencies and cache sizes. And, the Azure vCPUs too are "threads" and not "cores."

Moreover, something like this can happen and now the build time is almost half of what it used to be with basically no change on the Zephyr side: https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/

Unless we dedicate a specific physical "reference" machine (that can be made available for the next 10 years or something) for this purpose, I do not think tracking build time will be meaningful.

p.s. CPU performance isn't the only thing to consider in fact. Both network and storage performances do make a lot of difference as well.

[fabiobaltieri]

Yeah same feeling from me, this is very unpractical, too many things out of control. That said, if you feel like writing something to ingest the data from the action and store it long term just go ahead and do it, worst case we look at it two years down the road and say "yep, this is useless". I just would not put too much time on it.

we know building on windows is just a mistake in general

I want that on a t-shirt.