dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
687 stars 267 forks source link

Reduce F# real world benchmarks #4358

Closed psfinaki closed 1 month ago

psfinaki commented 1 month ago

Hey there, we've decided to reduce and disable real world F# benchmarks.

I am following the perf-auto-profiling issues and there are too many false positives and false negatives there that it's not possible to see a reliable trend. This is likely because parsing and typechecking are very complex and extensive workloads and it's probably too hard to get stable results even in the strictest environments like dotnet/performance has.

image

That said, 1) We'd like to keep the code compiling and running so that one day we can easily add some better real world scenarios. The project has a mechanism for syncing F# compiler version which is important. Hence the remaining benchmark is disabled.

2) We also have F# micro benchmarks in this repo, and they are more promising - we might add more of those soon.

psfinaki commented 1 month ago

I guess CI failures are unrelated?

LoopedBard3 commented 1 month ago

I guess CI failures are unrelated?

Yes, the CI failures are unrelated and are being hit on the main runs.

Another option for disabling these runs while keeping the current source would be to comment out or remove the runs from the yaml. Specifically https://github.com/dotnet/performance/blob/main/azure-pipelines.yml#L186-L200 and https://github.com/dotnet/performance/blob/main/azure-pipelines.yml#L488-L504. This will need to be done anyways to fully disable these test runs either way. Feel free to do it in your PR here and ping for review + merge, or let me know once this goes in so we can remove the yaml ourselves. Thank you!

psfinaki commented 1 month ago

@LoopedBard3 alright, sure, I disabled F# real world in the pipelines as well.

LoopedBard3 commented 1 month ago

@psfinaki this is looking good to me with the pipelines now disabled. Let me know once you want to merge it and I can complete the merge 👍.

psfinaki commented 1 month ago

@LoopedBard3 ready when you are, feel free to merge right away :)