Benchmarking (and testing) framework

[slizzered]

This could be split into 2 issues, but for development of ideas 1 issue might be sufficient now... both features will be closely related in code. (after doing a test, just record the time...)

It would be great to have some code that will a) test some defined input/output of the whole application (we had this before, but it was never really easy to see immediately if the results are the same or minimally different). Might be difficult since results are not 100% deterministic b) maybe also some unit-tests for smaller functions (mostly necessary if later some internal stuff will be improved/changed, like #51) c) create some fixed benchmarking scenarios (for whole simulation and also for single functions), so we have a fixed and reliable time value that we can use to measure performance changes that are introduced through new features. It would be good to have quite a list of parameter combinations. d) We should have a PERFORMANCE.md file that holds those values for different releases, so we can track the speed development over a longer timeframe

Things to compare in the future:

• run time for different numbers of Nodes, so we can evaluate load-balancing algorithms (do things like #23 actually work?)
  • does changing the network structure ( #17, #52 ) have any bad influence?
  • how much do we gain from K80 GPUs? (and future versions)
  • ... [please add more things]

[slizzered]

10 and #54 will be helpful to support the implementation of this issue

[erikzenker]

I had a look into some testing frameworks and finally decided to use Boost.Test. There were no big differences between the test frameworks, but since we are using boost anyways there is no reason to ship more dependencies with our code.

[slizzered]

very reasonable thinking

[erikzenker]

Just to give a reference to ongoing work of this issue.

[erikzenker]

We should get rid of a matlab based testing proceedure. Better would be some free available scripting language!