apjanke / octave-testify

New BIST (Built-In Self Test) functions for GNU Octave
GNU General Public License v3.0
4 stars 2 forks source link

fail-fast "PASSES M out of N tests" output is insufficient #75

Closed apjanke closed 5 years ago

apjanke commented 5 years ago

When you run plain test or test2, fail-fast behavior is the default.

But the non-verbose output for test2 in this case isn't sufficient; it doesn't make it clear that the test run was aborted.

>> test2 isosurface
PASSES 20 out of 21 tests
>>

That sounds like there were 21 tests total in the file, and only one of them failed. But there are actually at least 30 tests.

Add some additional output there: a line saying "test run aborted", and probably the detailed output for the failed test block.

apjanke commented 5 years ago

Better:

>> test2 isosurface
----- Failure details: block 25 -----
*****  testify.internal.BistBlock: error (index=25)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Y must match the size of V/
Code:

 y = -14:6:11;
 fvc = isosurface (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
Aborted test run due to test failure.
PASSES 20 out of 21 tests
>>
>> test2 isosurface -no-fail-fast
----- Failure details: block 25 -----
*****  testify.internal.BistBlock: error (index=25)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Y must match the size of V/
Code:

 y = -14:6:11;
 fvc = isosurface (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
----- Failure details: block 26 -----
*****  testify.internal.BistBlock: error (index=26)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Z must match the size of V/
Code:

 z = linspace (16, 18, 5);
 fvc = isosurface (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Z must match the size of V/, but got 'isosurface: X must match the size of V'
----- Failure details: block 33 -----
*****  testify.internal.BistBlock: warning (index=33)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/colors will be calculated, but no output argument to receive it./
Code:

 [f, v] = isosurface (val, iso, yy);
  -> success=0, msg=!!!!! error raised: isosurface: COL must match the size of V
PASSES 26 out of 30 tests (1 known bug)
>>