dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
694 stars 271 forks source link

Improvement to Output of ResultsComparer for Perf Report Analysis #2476

Open dakersnar opened 2 years ago

dakersnar commented 2 years ago

@mrsharm and I were discussing ways in which we could speed up the perf report process. We think generating the following links at the top of each entry to the table could speed up the process of investigating a specific benchmark. This extra data could be generated with a flag, so that the final report can exclude it. This would only be for analysis purposes.

  1. Link to a search of all dotnet/runtime issues with that specific test as a query. For example, https://github.com/dotnet/runtime/issues?q=is%3Aissue+System.Memory.Span%3CByte%3E.Fill%28Size%3A+512%29
  2. Link to a search of all dotnet/perf-autofiling-issues issues with that specific test as a query. For example, https://github.com/dotnet/perf-autofiling-issues/issues?q=is%3Aissue+System.Memory.Span%3CByte%3E.Fill%28Size%3A+512%29
  3. Link(s) to the historical perf lab data for this test. This requirement is a little bit trickier, as providing an individual link for every config would add too much text to the perf report and make it unreadable. We brainstormed a solution in which only two or three common architectures are linked, to give the perf report writer a starting point. For example, we could provide links to Windows x64, Windows Arm64, and Ubuntu x64:

What would be ideal is if we could provide one link, for a specific benchmark, that has links to each configuration. Basically a reversal of how the data is currently indexed https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/TestHistoryIndexIndex.html. @DrewScoggins, do you know how difficult it would be to be able to query this data in reverse (benchmark -> architecture instead of architecture -> benchmark)?

dakersnar commented 2 years ago

@mrsharm has started the process of creating this tooling as an interactive VSCode notebook. I'm going to use this issue to track some TODOs we have to augment the tooling.