Pinpoint non-Telemetry test support

dave-2 commented 6 years ago

Looks like there's a bunch of differences between Telemetry and non-Telemetry tests.

Different parameters and parameter validation (benchmark/story vs gtest)
Different Swarming/Isolate arguments (--story-filter, --gtest-filter, etc.)
Different output format (Chart JSON vs Graph JSON)

Here's my proposed solution to abstract out the differences:

Have different RunTest and ReadValue Quests for Telemetry and non-Telemetry tests. They can inherit from the same parent classes in order to share code.
Add a test_config field to the Job as a PickleProperty. TestConfig will take care of storing and validating the test parameters. It will also generate the Quest list, allowing us to use the different Quests.

@perezju @simonhatch @anniesullie @eakuefner

perezju commented 6 years ago

Plan sgtm

simonhatch commented 6 years ago

sgtm

CalebRouleau commented 5 years ago

I can't figure out how to run a Pinpoint job against a non-Telemetry benchmark. The docs here https://chromium.googlesource.com/chromium/src/+/master/docs/speed/perf_trybots.md seem Telemetry-specific.

The UI here https://pinpoint-dot-chromeperf.appspot.com/

asks for a "benchmark" and a "story" which may not exist for non-Telemetry benchmarks. Furthermore, when I guess what to do and put in the non-Telemetry benchmark executable name, it does not work and the logs show that it is using Telemetry's "run_benchmark" tool.

I'm not sure what else to do to run a non-Telemetry benchmark. Is there a supported commandline tool?

catapult-project / catapult

Pinpoint non-Telemetry test support #3729