Closed psfinaki closed 1 month ago
Monthly update :)
As of now, things got stable here - the FCS is auto-updated, issues are created and so on. I merged the results into a dashboard.
Now, we have some "real world" benchmarks and, more recently, micro benchmarks. All of them are copied from our repo.
As of today, I deem real world benchmarks useless, times go up and down, we get these "improvements" and "regressions" but I think it's all false. I will probably just delete them soon to decrease the noise.
Micro benchmarks - I am still evaluating that. They look better:
There is no avalanche of false alarms, in fact, there are none yet - we get notifications about benchmarks getting executed but not about regressions or improvements. I am basically waiting for a ticket like this to arrive to see if this is anything reasonable or not.
Update on this - the above still proved to be the case, the real world benchmarks are full of false regressions and improvements. Hence I decided to disable them here, keeping the code and infra so that we can add some better real world benchmarks in the future if we want.
I will close this issue, but I am opening this one about micro benchmarks because they seem to be more promising and worth experimenting.
Currently FCS version is hardcoded in the dotnet/performance repo so we are unable to track performance regressions using it. See if we can easily plug in the main version of FCS there.