catapult-project / catapult

Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..
https://chromium.googlesource.com/catapult
BSD 3-Clause "New" or "Revised" License
1.91k stars 563 forks source link

[📍] Fail the Telemetry test if nothing ran #4527

Open dave-2 opened 6 years ago

dave-2 commented 6 years ago

If a benchmark is disabled or the story filter doesn't match any stories, Telemetry produces a "passing" return code. We want to force this to be a failure, so it will be easier for users to diagnose what went wrong. In Pinpoint, the current behavior causes the "run test" step to pass, but the "read values" step to fail, which is a red herring.