Begin unit testing framework

[dondi]

Start building a test suite for GRNmap. Start by learning MATLAB's framework for it (http://www.mathworks.com/help/matlab/matlab_prog/write-simple-test-case-with-functions.html). Once you are comfortable with running the samples, start by writing tests against the actual GRNmap code. Focus on pure mathematical computations first; we will eventually branch out to things like file validation etc.

• Create a sub-branch of beta called test_suite to initially isolate the unit test code while we are figuring it out
• Get sample data from @bengfitzpatrick to form our test fixture(s)
• Probably the simplest initial function to test is forward simulation
• Once a suite is up and running, we can merge the code into the main development branches (beta, master)

[trixr4kdz]

Created sub-branch

[dondi]

Immediate goals to be done by March 18:

• Commit test files to test_suite branch (even if just tutorial or samples)
• Set up/determine a convenient way to run the full test suite
• Try to write a real-world test for the forward simulator, consisting of:
  1. Opening a particular file
  2. Running the forward simulator
  3. Examining the resulting output file

[trixr4kdz]

Added test files to test_suite branch

[trixr4kdz]

(entered by @dondi typed this)

During our 3/18 meeting, we were able to write a real-world test. It has a single assert so far and only tests one file, but it is a genuine start.

Tasks under this issue from this point:

• Start fleshing out more tests (i.e., verify that the entire file was read correctly; read additional files; start testing for proper handling of errors in files, etc.)
• Set up/determine a convenient way to run the full test suite (this increases in importance as the number of tests and test files grows)
• Figure out how to keep test code in the _testfiles directory while still correctly invoking the code in the matlab directory. For simplicity at the moment, the tester .m files are located in matlab. However, in the long run, it is better to keep them in a separate location.

[dondi]

@trixr4kdz discovered the addpath function; for replicability it needs to use relative paths. Here is a reference, and also based on the file structure the path needed looks like it should be ../../../matlab:

http://www.mathworks.com/matlabcentral/answers/4701-how-to-manage-relative-addpath-uses-in-subfunctions-when-using-svn-for-example

[dondi]

Next task is to finalize the file structure using addpath and calltests, after which tests can now just be churned out.

[dondi]

After structure is finalized, work can be merged into beta and eventually into master.

[trixr4kdz]

Added more test cases. Fixed file paths so that we can keep test cases in test_files folder instead of matlab folder. Also, removed warning messages from test cases. Still need to make more test cases

[kdahlquist]

Let's discuss this at our next meeting in the context of an actual workflow for testing related to #74 and #84. I'm thinking that we might be able to get Tessa involved in working with @trixr4kdz in doing some testing tasks.

[kdahlquist]

Merge test-suite branch into beta branch and all further development along these lines should be in beta.

[kdahlquist]

In the next week, @tessaam and @trixr4kdz should work together on automating tests that @tessaam has completed manually (e.g. issue #74 and others).

• For the small networks, @tessaam has observed no differences in the output files from multiple runs (i.e., all differences were 0), so it should be straightforward to write the test.
• For the large networks @tessaam has observed very small differences (out in the far decimal places). We need a criteria for saying that there is "no difference" between the results of the tests.

[kdahlquist]

• @trixr4kdz merged test_suite branch to beta last Monday.
• the 16 input test files of the small networks are now compatible with the changes to beta
• reran to get the corresponding outputs
• wrote tests
  • all output sheets produced as expected (e.g., if fix_b=0, then optimized_threshold_b sheet should be made)
  • wrote a test to change the name of the optimization_diagnostics.jpg file and then fixed the code #106

[trixr4kdz]

@dondi or @bengfitzpatrick just a quick question, even when I make tests for the individual functions, I basically still have to run the entire program so the new output will overwrite the past output that I'm trying to compare with. Also, that would mean that running all 16 tests will take very long overall. Is there a way to make running the tests faster and not overwrite the testing files or is this exactly how its supposed to be? Currently, the tests have been split up into the individual functions they are trying to test (i.e. readInputSheetTest is testing readInputSheet)

[trixr4kdz]

Possible solutions that @kdahlquist proposed to prevent overwriting files (that we will have to implement anyway in the future):

• (This is more likely what we want) append a number to files that have similar names (i.e. if we have a file called 4-genes_6-edges_output.xlsx and we run the program again, we should get 4-genes_6-edges_output(1).xlsx)
• OR open a dialog box to ask user where they want to save the output files

About running GRNmap in parallel, we need to take advantage of the multi-core computers and assign individual runs to each core. I will need to ask Grace how to do this.

[dondi]

From a general architecture perspective, the ideal solution is to actually intercept the data before they get written out. Unless streaming of some sort is happening, the actual results should be in memory initially, with the writing to a file being a last step. A unit test should be able to receive the data before it is written to a file. This test would verify that the computed data were indeed wat was expected.

With that separation, the actual writing to a file can then be tested by itself, in isolation from the computation. One would prepopulate the data before it is written out, then write out to a temporary file (thus avoiding the overwrite issue), and finally assert that the file content does correspond to the in-memory data.

If achieving this separation will involve too much refactoring at the moment, then the numbered-files approach may be the intermediate solution. But in the long term we should aim for this separation of data computation from data output to a file.

One last thing: many language platforms have a "generate temporary file" function. If MATLAB has this, we can use that approach for the test file naming and handling. One advantage of temporary files is that the system auto-deletes them for you when done.

[kdahlquist]

We will verify at the meeting today whether or not we can close this issue. It seems that we need to open separate issues dealing with @dondi 's comments:

• having the testing suite intercept the data before it is written, or
• creating a temporary file
• appending a number to successive versions of an output file so that the last file doesn't get overwritten (this latter one is definitely an issue we need to break out for general functionality).

[trixr4kdz]

The items listed in the comment above are now separated into different issues. See #135 and #136. Also, #134 describes using setUp and tearDown methods for more efficient test runs.

[kdahlquist]

Since the individual issues are now broken out, we are going to close this issue.