[C++][Benchmarking] Measure and trace build times

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

https://arrow.apache.org/

Apache License 2.0

14.67k stars 3.56k forks source link

[C++][Benchmarking] Measure and trace build times #36091

Open pitrou opened 1 year ago

pitrou commented 1 year ago

Describe the enhancement requested

To avoid accumulating a technical debt of increasing build times, stifling contributor productivity, we should measure build times over time and warn of significant regressions (related: issue #36090).

Here is a temptative list of requirements:

build in debug mode (building with optimizations and without debug information is rather rare when developing)
build shared libraries (there is no strong use case for static libraries, especially when developing)
build with tests enabled
have all non-small dependencies pre-built, not bundled (from conda-forge perhaps?)
enable as many Arrow components as possible
if some components cannot avoid building a large dependency on-the-fly, disable them

(by large dependency, I mean protobuf, gRPC, libcurl, OpenSSL, AWS SDK, etc.)

Component(s)

C++

pitrou commented 1 year ago

cc @jonkeane @westonpace @assignUser

pitrou commented 1 year ago

Also @lidavidm FYI.

assignUser commented 1 year ago

CDash (CMake/CTest dashboard) has a build time graph but that is rather simplistic. Could we potentially read the ninjalog and extract some compact data to add to conbench via a cusom field or somethign like that cc @austin3dickey

pitrou commented 1 year ago

I was really thinking something simpler, such as time ninja ....

assignUser commented 1 year ago

Sure but we would still have to store that somewhere to be able to detect regressions.

jonkeane commented 1 year ago

some compact data to add to conbench via a cusom field or somethign like that

I don't think there even necessarily needs to be a custom field for this in conbench, it can be added just like any of the other measures of performance (albeit it'll be from a different kind of run than the others, but that's totally fine!).

If someone has the operative parts of the benchmark (e.g. a CI job that does the required bits that Antoine mentioned, I would be happy to spend some time wiring that up to send data to conbench with the conbench tools/utilities we have).