dotnet / fsharp

The F# compiler, F# core library, F# language service, and F# tooling integration for Visual Studio
https://dotnet.microsoft.com/languages/fsharp
MIT License
3.87k stars 779 forks source link

`Timebox`: Set up FCS sync in the dotnet/performance repo #16639

Closed psfinaki closed 1 month ago

psfinaki commented 7 months ago

Currently FCS version is hardcoded in the dotnet/performance repo so we are unable to track performance regressions using it. See if we can easily plug in the main version of FCS there.

psfinaki commented 3 months ago

Monthly update :)

As of now, things got stable here - the FCS is auto-updated, issues are created and so on. I merged the results into a dashboard.

Now, we have some "real world" benchmarks and, more recently, micro benchmarks. All of them are copied from our repo.

As of today, I deem real world benchmarks useless, times go up and down, we get these "improvements" and "regressions" but I think it's all false. I will probably just delete them soon to decrease the noise.

image

Micro benchmarks - I am still evaluating that. They look better: image

There is no avalanche of false alarms, in fact, there are none yet - we get notifications about benchmarks getting executed but not about regressions or improvements. I am basically waiting for a ticket like this to arrive to see if this is anything reasonable or not.

psfinaki commented 1 month ago

Update on this - the above still proved to be the case, the real world benchmarks are full of false regressions and improvements. Hence I decided to disable them here, keeping the code and infra so that we can add some better real world benchmarks in the future if we want.

I will close this issue, but I am opening this one about micro benchmarks because they seem to be more promising and worth experimenting.