mariuszhermansdorfer / SandWorm

Augmented Reality Sandbox for Grasshopper & Rhino
MIT License
20 stars 11 forks source link

Add console app for running isolated benchmarks #32

Closed philipbelesky closed 4 years ago

philipbelesky commented 4 years ago

This adds a console app to the project that can run benchmarks outside of Rhino/GH over a set number of iterations. It does so by mocking parameters within the SandWorm component that are necessary to simulate acting on a mesh. The idea here is to be able to easily A/B changes without running Rhino/Grasshopper and to measure performance more reliably.

Currently this relates to #29 and #30 so the test functions and overall classes are setup to test the mesh coloring stage. If it is useful for optimising other parts of the code it should be able to be extended to cover different mock data/methods and the amount of mock data setup could be reduce by creating a shared class that is inherited by both SandwormComponent and MockMesh.

I'm a little split as to whether this should be in the main repo or kept on a branch. The former is a bit less clean, but is more convenient. If it proves useful it would also be easier to be able to improve the console app itself along the rest of the code. We could also just remove it later if it ends up being less useful?

Also a result of this, the console app needs to reference a Sandworm.dll so Sandworm needs to be built first (ideally as the released build) and the 'erase' step of the build event has been removed.

philipbelesky commented 4 years ago

Oh, also, a quick note if testing this: the solution file was moved into the base directory, which means the packages folder also needs to move up a level.

mariuszhermansdorfer commented 4 years ago

Let's add it to the master for convenience and remove later when no longer needed

mariuszhermansdorfer commented 4 years ago

Great work on the console app @philipbelesky! It seems that there is a minor issue with the way times are measured, though. Currently the first test seems to suffer from a small overhead of approx. 6 ms. Easy to reproduce through reversing the order of tests in the list. image image

philipbelesky commented 4 years ago

Hmm, I noticed a much smaller discrepancy (sub 1ms avg) when running which seemed enough to ignore when profiling bigger functions. But I was mostly benchmarking within a VM which is probably a less typical environment.

However, it turns out the mocking class means that the benchmarks are now compatible with using benchmark.net now, so I've switched over to using that in 2381ef599a71a5f75af4207b2ca80f20a025f714. They take a little longer to run but the results are more consistent and all the timing code can be removed in favour of an attribute marker.

Screen Shot 2019-09-15 at 11 35 23 am