Add regression testing

[mk306]

1.Add regression testing features to the application... Figure out what and how should be checked. Cant really check gui.

[mk306]

An overview...

1.Come up with an objective oriented method of testing of major subcomponent of MLO

• Report if results substantially differ from the back data
• Back data does not have to be matched exactly!
• Including new tests could be done by providing a test script, with an explanation of what it is testing, and some sort of result processing script. For example testing PSO could be a test of 30 run mean of 10 different trails. If the mean increases substantially... report an issue.
  1. System outline
  2. Build a script which is going to create an html website containing a summary of all the tests
     • Set up a crone deamon ( It can run on my machine) which on daily basis would run the tests and email the results to specific users
     • The test should check-out the data from git repository and run the tests.
     • It should be very easy to add new tests.
     • specific tests could include emails of users which should be informed about particular results
     • Set up the testing commands in a make file, so it would be easy to integrate with job scheduling process as well as give the possibility of manual testing.

[mk306]

Tests cases that should be included

1.When we use a classifier if we do not find at least a single example of a valid design svm is going to classify whole of the design space as invalid... we need to check that we tackle this issue accordingly. The simplest test case would be a fitness script which classifies ~10-15% of the design space as valid.. 2.Use a set of tests to evaluate performance.. few artificial functions - for continuous spaces reconfigurable radio + anson - for discrete spaces 3.Use a set of artificial test with 3-10 dimensions... to check if multi-dimensional problems are tackled properly. 4.Create an artifical function with one high peek, and a flat landscape... this is important to check that we handle stagnation properly :) Create both a discrete and continous version 5.Classifier does not need to be tested currently - we should do it at some point though... 6.We should test the regressors for performance... execution time, memory profile, and accuracy... Please discuss this with me as it it's not easy to define... 5-6 functions would be good...

More to come :)

[mk306]

7.Scalability of the regressors is very important! Check for 10-30-100-150 examples