The current output of benchmarks allows to follow to total time and memory usage over time, but does not really indicate the evolution of sub-part of the benchmark scripts.
In some cases, this might completely hide some trend. For instance, the spectrum_1d CPU time is dominated by the data reduction step, but the fit or flux points parts can evolve very significantly.
We need a system able to track the various sub-tasks independently and able to provide a easy reading of results.
Additionally we might want to add some profiling information in order to better track performances degradations.
We have discussed using asv in the past. We might reconsider it.
@QRemy has already tested some profiling solutions and might want to comment. Opinions @adonath , @Bultako , @bkhelifi ?
The current output of benchmarks allows to follow to total time and memory usage over time, but does not really indicate the evolution of sub-part of the benchmark scripts. In some cases, this might completely hide some trend. For instance, the spectrum_1d CPU time is dominated by the data reduction step, but the fit or flux points parts can evolve very significantly.
We need a system able to track the various sub-tasks independently and able to provide a easy reading of results.
Additionally we might want to add some profiling information in order to better track performances degradations.
We have discussed using asv in the past. We might reconsider it.
@QRemy has already tested some profiling solutions and might want to comment. Opinions @adonath , @Bultako , @bkhelifi ?