search

dotnet / linker

388 stars 127 forks source link

Perf regression / stress tests #2978

Open jtschuster opened 2 years ago

jtschuster commented 2 years ago

We currently don't have any tests to make sure we don't have large performance regressions on size or speed, which could have caught https://github.com/dotnet/linker/issues/2975 before it became a blocking issue for xamarin.

Linking a xamarin application, runtime libraries, and another large application in rolling builds could be a good start.

Any other ideas or thoughts?

mrvoorhe commented 2 years ago

At Unity we have size and performance test for our linker.

For size we have a set of sample projects. We maintain "baseline" file sizes for our main branch that update nightly. For PR's we run the tests and compare to the baseline. Any file sizes that regress by more than a certain amount result in a failure.

We don't have an automated way to stop performance regressions from landing. But we do record the performance nightly and have a graph of the performance over time that we review from time to time. This has been good enough to catch major regressions.

Sadly we are well behind upstream main these days. So this doesn't help you in practice :smile: Just thought I'd share what we do. Sounds like you are thinking of doing something similar.

MichalStrehovsky commented 2 years ago

We have a perf lab - we could set up throughput runs there. It results in bugs with nice graphs like this: https://github.com/dotnet/runtime/issues/51006. Onboarding illink shouldn't be too hard. IIRC getting crossgen2 onboarded was pretty quick.

They're doing stuff for size too, so at least indirectly, ILLink is covered through Blazor WASM: https://github.com/dotnet/runtime/issues/55608

I suggest talking to the perf team - Drew (who filed the above issue) might be a good starting point.

jtschuster commented 2 years ago

Simple benchmarks to run linker on a Hello world app are here: https://github.com/dotnet/performance/pull/2581. It's currently blocked by an issue with the SDK looking for an unreleased version of .net6 libraries (https://github.com/dotnet/sdk/issues/27888).

jtschuster commented 2 years ago

If we want to set up our own pipeline to run perf tests on every commit, here are some examples / things to look at:

runtime/perf.yml at main · dotnet/runtime (github.com) - This includes a bunch of other yml templates that are used for the other various pieces runtime/microbenchmarks.proj at main · dotnet/runtime (github.com) - This is what does the work to create the Helix Workitems to send to Helix for execution. runtime/performance-setup.ps1 at main · dotnet/runtime (github.com) - This also does a bunch of work to build out all of the components required for a run in helix. ("Performance windows x64 release coreclr JIT micro perftiger NoJS False net7.0" job in dotnet-runtime-perf pipeline) - This is a link to one of our pipeline runs. You can see what we are doing, and should be able to match this up with the yml. The big things to see are all of the handling of the product, performance setup, ci setup, and Send To Helix. ("Performance windows x64 release coreclr JIT micro perftiger NoJS False net7.0" job in dotnet-runtime-perf pipeline)