Open vzarytovskii opened 2 years ago
Nice to see this getting traction 👍
FYI I created https://github.com/safesparrow/fsharp-benchmark-generator for automating FCS testing. I was planning to add more examples there and make it more robust, but not much code there at the moment.
I'd be more than happy to help either extend that or work on an equivalent, more official tool. I can also transfer the repository if that'd be of any use.
I think in general a separate repository with benchmarks is better than putting them inside the main repository - similar to the approach taken in dotnet/performance. Not sure if all three types of tests (VS, FCS code analysis, FCS runtime) should live together or not. Maybe they could share a library with utilities (eg. for code/project generation) but be separate pieces of code & infrastructure?
Also I think it would be nice if whatever benchmarking tools are created are available for local runs and not just CI.
I created some parsing and type checking benchmarks on FSharpPlus and FsToolkit.ErrorHandling which take reasonable amount of time but should also hopefully catch most regressions.
https://github.com/dotnet/performance/pull/2592
They can be run locally, with local build of FSC plugged in via project file. I'll think about making that easier somehow.
@0101, given that your PR in dotnet/performance has been merged for 2 weeks now, can we see some results, graphs? Does it run regularly yet?
@kerams it should be running now, and data being collected. Still have to figure out how to get to them.
Ping
Ping
Latest results are from November last year, it seems it doesnt run anything, and no alerts were produced. We're mostly using local machines not run before/after.
No data for other (non-F# types of runs either). Not sure if it was migrated somehwere @DrewScoggins?
We need to come up with reliable FCS benchmarks, and run them as part of
dotnet/performance
, ideally every release (on every signed build for current release branch).Some ideas for scenarios we want to test:
Some metrics we are interested in: time spent in scenarios, allocations (how much do we promote, is there anything ending up in LOH, etc), time spent in GC, etc.
More info and context: https://github.com/dotnet/fsharp/discussions/12526 & https://github.com/dotnet/fsharp/pull/13419