Deprecated Catapult GitHub. Please instead use http://crbug.com "Speed>Benchmarks" component for bugs and https://chromium.googlesource.com/catapult for downloading and editing source code..
Pinpoint has deep integrations with many parts of the perf waterfall and Chrome infrastructure. We should run regular integration tests to spot when changes to the perf waterfall break Pinpoint.
We'll likely want to cover every configuration. Launching a Swarming job to run one iteration costs about 1 minute. So assuming 16 bots, (1 minutes * 10 repeats) / (1 hour * 16 bots) == 1.0%. The overhead is about 1% of our total bot capacity.
If we were to also test builds as part of the integration tests, we'd consume a significant portion of our build capacity. So we should use the latest prebuilt from the perf waterfall. We will need some way to keep track of what that is.
@nedn @simonhatch @perezju
Of course, you could argue that spotting breakages after the fact is too late. Especially if the compatibility-breaking changes are in Chromium, they would be permanently in the revision history, so any bisects across that revision range would break, leading to fallout for multiple months. We also need some process/plan in place for figuring out how to prevent breakages.
Follow-up from #4534.
Pinpoint has deep integrations with many parts of the perf waterfall and Chrome infrastructure. We should run regular integration tests to spot when changes to the perf waterfall break Pinpoint.
We'll likely want to cover every configuration. Launching a Swarming job to run one iteration costs about 1 minute. So assuming 16 bots,
(1 minutes * 10 repeats) / (1 hour * 16 bots) == 1.0%. The overhead is about 1% of our total bot capacity.If we were to also test builds as part of the integration tests, we'd consume a significant portion of our build capacity. So we should use the latest prebuilt from the perf waterfall. We will need some way to keep track of what that is.
@nedn @simonhatch @perezju
Of course, you could argue that spotting breakages after the fact is too late. Especially if the compatibility-breaking changes are in Chromium, they would be permanently in the revision history, so any bisects across that revision range would break, leading to fallout for multiple months. We also need some process/plan in place for figuring out how to prevent breakages.