search

catapult-project / catapult

Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..
https://chromium.googlesource.com/catapult
BSD 3-Clause "New" or "Revised" License
1.93k stars 563 forks source link

Write an apps script to inform benchmark owners of recent perf regression bug changes. #4311

Closed anniesullie closed 6 years ago

anniesullie commented 6 years ago

One big worry I have is that it's hard to know if a bug may have been WontFix-ed incorrectly (every once in a while I see a bug WontFix-ed because the CL author thinks the cause is not their CL, and sometimes they disagree with the benchmark owner on severity). It would be really easy to write an apps script to send a weekly report to each benchmark owner that lists all bugs WontFix-ed that week.

I thought that we may as well also have lists of Fixed and open bugs as well, since they are also easy and the benchmark owner may want to see them.

This collides a bit with benchmark health report, but is at a narrower scope (just making sure owners are able to follow up on bugs if they wish). @nedn @zeptonaut are you okay with me trying this out?

nedn commented 6 years ago

I don't see any problem with this. But I think we should try this out with a small audience first?

@zeptonaut maybe should work closely with you on this so he gain expertise in this area as well

anniesullie commented 6 years ago

Sure. Do you have any recommendation for small audience, or maybe just ping benchmark owners and see who writes back?

nedn commented 6 years ago

I think we can first start with memory benchmark & system_health common benchmark. They generate most volume of alerts anyway.

anniesullie commented 6 years ago

It might be good to also have a smaller benchmark owner on the list. It's possible we'll want a different format for benchmarks that get > 20 bugs per week vs benchmarks that get 0-2.

nedn commented 6 years ago

SGTM

zeptonaut commented 6 years ago

Sorry: somehow I missed this in the perf season. This sounds like a good plan to me, but I agree that rolling it out to a small audience would be a good start.

anniesullie commented 6 years ago

I wrote an apps script and documented the output here: https://docs.google.com/document/d/1q5fTGpKDPvA2m2SMy2XcIwAr10K4i6tsSh-M-umPfTI/edit?ts=5ab90908#

Got good initial feedback from benchmark owners; planning to turn this on today with a survey link at the bottom to ensure there's a clear feedback loop if the email isn't helpful.

anniesullie commented 6 years ago

Script was sent to benchmark owners with good feedback; scheduled to update weekly.