Feature request: Make it easier to get traces for bisect

[petrcermak]

When there's a memory regression and the bisect blames a patch, we want to know why the particular memory component regressed. Unless there's something obviously wrong with the CL, the only way to move forward is to analyze the traces before and after the blamed patch.

However, it is quite complicated to get the traces at the moment. For example, given bisect results (https://bugs.chromium.org/p/chromium/issues/detail?id=633182#c17), you have to do the following:

1. Click on the "Buildbot stdio" link (e.g. http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3165)
2. For the "before" (without blamed patch) and "after" (with blamed patch) revisions:
   1. Figure out which step corresponds to the revision. This is quite difficult if either of them is at either end of the bisected range because both the "good" and "bad" revisions are tested under the "Gathering reference values" step, but there are no labels marking which of the "Performance Test M of N" sub-steps correspond to which of the two revision.
   2. For each "Performance Test M of N" sub-step (remember that the bisect repeats each revision N times):
      1. Open the "stdio" of the sub-step (e.g. https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3165/steps/Working%20on%20revision%20chromium%40408626.Performance%20Test%201%20of%205/logs/stdio)
      2. Search for the line with the URL of the regressed page (e.g. @@@STEP_LOG_LINE@Captured Output@View generated trace files online at https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_0-2016-08-08_16-15-21-69363.html for page https://www.facebook.com/rihanna@@@).
      3. Open the trace URL (e.g. https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_0-2016-08-08_16-15-21-69363.html)

To sum up:

1. There's no way you'll find the traces unless you're very familiar with the infrastructure.
2. Even if you are familiar with the infrastructure, retrieving them is very tedious.
3. There's no way for non-Googlers to get the traces.

As a result, this adds a lot of unnecessary work for sheriffs.

Is there any way this could be automated? Ideally, the bisect would include links to the traces when it updates the bug. Alternatively, the bot could have a dedicated step that just lists links to the traces for each of the tested revisions and post a link to this step when it updates the bug (alongside "Buildbot stdio" and "Job details").

@anniesullie @ro-berto

[anniesullie]

@dave-2 here is another bug at the intersection of perf dashboard and bisect. The trace urls are in the chartjson; I think we could send those from bisect->perf dashboard in the call to /post_bisect_results. Picking the best trace for the common case where the summary metric is bisected will likely be tricksy, maybe @eakuefner has advice?

[petrcermak]

Couldn't we just always send all relevant traces? E.g. if you bisect a value that's an aggregate of pages pA, pB and do 2 repeats, then I think (links to) traces for all of pA1, pA2, pB1, pB2 should be provided.

[natorion]

I think this is already fixed? I have seen the option to download the traces.

[simonhatch]

I think the option to view the trace is only available via perf-try jobs. Bisect jobs don't parse the output for the cloud links or surface them to the dashboard in any way.