search

mcellteam / nutmeg

nutmeg is a concurrent regression test framework for the MCell simulator
BSD 2-Clause "Simplified" License
1 stars 1 forks source link

Report invalid tests in summary #2

Closed jczech closed 8 years ago

jczech commented 8 years ago

Here's what a nutmeg summary looks like:

-------------------------------------
Ran 7 tests in 0.122240 s:  SUCCESSES[7]  FAILURES[0]

This is pretty useful, but, if you're running the full testsuite (currently 171 tests), then you might miss invalid tests, because invalid tests are only reported at run time like this:

warning_msg_negative_dc                     ::   FILE_MATCH_PATTERN              [SUCCESS]
warning_msg_high_rxn_prob                   ::   FILE_MATCH_PATTERN              [SUCCESS]
warning_msg_negative_rxn_rate               ::   FILE_MATCH_PATTERN              [SUCCESS]
2016/01/15 15:39:01 Error parsing test description in /home/jacob/nutmeg/tests/warning_msg_surface_full: invalid character '(' in string escape code
warning_msg_negative_rxn_rate_timevary      ::   FILE_MATCH_PATTERN              [SUCCESS]
warning_msg_output_step_long                ::   FILE_MATCH_PATTERN              [SUCCESS]
warning_msg_output_step_short               ::   FILE_MATCH_PATTERN              [SUCCESS]
warning_msg_sm_list_rel                     ::   FILE_MATCH_PATTERN              [SUCCESS]

It might be nice to provide a summary of all invalid tests like this:

-------------------------------------
Ran 7 tests in 0.122240 s:  SUCCESSES[7]  FAILURES[0]
Some tests could not be run: INVALID[1]
haskelladdict commented 8 years ago

With commit a82d6e66 nutmeg should now properly report errors encountered during preparation of tests (such as parsing test description JSON files are creating directories).