search

voltrondata-labs / arrow-benchmarks-ci

Benchmarks CI for Apache Arrow project
MIT License
0 stars 5 forks source link

Stop benchalerts from alerting on test-mac-arm and languages other than Python/R #142

Closed austin3dickey closed 1 year ago

austin3dickey commented 1 year ago

We used to not alert in those cases, but #132 turned them back on again. Stop them.

Relates to https://github.com/conbench/conbench/issues/475

alistaire47 commented 1 year ago

We should really make a plan for the future of alerting for cases we're not alerting on—if we're not telling anyone when they fail, why are we running them? (There's still some value to seeing when they fail, but that's just more testing, not actual benchmarking for regressions.) Is turning alerting on pending improvements in data quality? Do we need to talk with Arrow devs to make a plan for these?

Whatever the plan, we should have an issue for it. I'm happy to open one, though if someone else has more preexisting context for the situation, they could lay out the situation better.

austin3dickey commented 1 year ago

The original rationale for turning off test-mac-arm alerts was here: https://github.com/voltrondata-labs/arrow-benchmarks-ci/pull/109

Agreed that we should have a plan for reducing noise in these benchmarks that leads to turning the alerts back on. A big-impact solution here could be https://github.com/conbench/conbench/issues/635.