The custom stopping criterion introduced in https://github.com/NVIDIA/nvbench/pull/151 currently triggers linear regression and R2 computation every second is_finished() call. Per @julie-is-late comment:
depending on the expected amount of time a test can take, i.e. if a test is very fast when compared to the amount of time to compute entropy, it might be better to calculate entropy and check if you're done at even bigger steps. This could even then be made "smart" by taking into account: how far is the current slope from the goal angle? what is the cuda_time of each run?
The custom stopping criterion introduced in https://github.com/NVIDIA/nvbench/pull/151 currently triggers linear regression and R2 computation every second
is_finished()call. Per @julie-is-late comment:We should experiment with this idea.