search

clean-development / engine

Game Engine
Apache License 2.0
2 stars 3 forks source link

Benchmarking Utility #16

Open smudge202 opened 8 years ago

smudge202 commented 8 years ago

One or more of our issues (such as #7) explicitly request we perform some benchmarking so we can track the performance of components. So as not to hold up development on the other components, I'm transferring those requirements to this issue.

My intent is to create a Command Line utility that can spool up a dedicated azure VM which can perform stress / performance testing in order to ascertain benchmarks. I'll create the necessary directories in this repository so that changes to component systems and associated benchmarks can be stored alongside the changes.

I'm probably going to use Compose to do this, so I'll drive that package towards completion of the 1.0.0-beta this week. Hopefully that way only benchmarking will be blocked by Compose.

We need to evaluate and potentially implement one of the following two technologies (open to additional options if anyone knows of one):

smudge202 commented 8 years ago

https://github.com/Microsoft/xunit-performance/

smudge202 commented 8 years ago

So I had a look at the MS Performance tools. I've not got as far as actually utilising them yet because the packages are not (from what I can see) published to NuGet.

I've raised an associated issue here:

I contemplated temporarily naughtily hosting the packages on an appveyor account feed if the packages worked and adding the respective NuGet.config to the engine so that collabs on this project could be given permission to access them. However, @pharring has pointed out the packages already exists on a public myget feed (https://github.com/Microsoft/xunit-performance/issues/100#issuecomment-154520619).

Good thing, because I'm not sure appveyor have read-only permissions available on account feeds yet (https://github.com/appveyor/ci/issues/35).

I'll continue checking out the MS benchmarking as the MS fanboy I can be, but there are alternatives available: https://github.com/PerfDotNet/BenchmarkDotNet

smudge202 commented 8 years ago

My write-up on BenchmarkDotNet:

https://github.com/PerfDotNet/BenchmarkDotNet/issues/33#issuecomment-154549833

No clue what's wrong; I'm going to assume it's trying to pull information out of the app domain in such a way that it doesn't work through DNX.

As for the MS variant, the current standard xunit runner doesn't detect the tests, and we can't add the dnx runner MS ships because it requires dnxcore50, which for one reason or another is unavailable from dnx46 which the test project targets.

Not going well.