Pinpoint interprets test failure as 'significant difference'

catapult-project / catapult

Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..

https://chromium.googlesource.com/catapult

BSD 3-Clause "New" or "Revised" License

1.91k stars 563 forks source link

Pinpoint interprets test failure as 'significant difference' #4483

Open schuay opened 6 years ago

schuay commented 6 years ago

See https://pinpoint-dot-chromeperf.appspot.com/job/14b1d1dc240000.

The build for [oilpan] Cleanup tracing scopes by mlippautz@chromium.org fails, but the failure is interpreted as a significant difference.

benshayden commented 6 years ago

@dave-2

dave-2 commented 6 years ago

Thanks for the report! I'll be working to improve the wording for test failures during performance bisects in the coming weeks.