search

lorenzwalthert / touchstone

Smart benchmarking of pull requests with statistical confidence
https://lorenzwalthert.github.io/touchstone
Other
53 stars 7 forks source link

Make indicator boundary customisable #106

Open assignUser opened 2 years ago

assignUser commented 2 years ago

I just looked at https://github.com/kgoldfeld/simstudy/pull/129 And I wondered if the users could be a able to specify a threshold himself for the icon. Does it always make sense to check if 0 is contained? One could argue that a ci that contains a 1% performance drop is still not enough to justify a 🐌 icon, so if the user specifies a custom value for null_boundary (Better name needed I think ), we check if the confidence interval overlaps with it.

Originally posted by @lorenzwalthert in https://github.com/lorenzwalthert/touchstone/issues/71#issuecomment-1015322809

assignUser commented 2 years ago

I agree, that would be good, maybe something like max_performance_loss or something.