apjanke / octave-testify

New BIST (Built-In Self Test) functions for GNU Octave
GNU General Public License v3.0
4 stars 2 forks source link

Test failures not in octave's __run_test_suite__ #72

Open apjanke opened 5 years ago

apjanke commented 5 years ago

Testify is failing more tests than core Octave is.

Octave 4.4.1:

Core __run_test_suite__:

Summary:

  PASS                            14933
  FAIL                                7
  XFAIL (reported bug)               54
  SKIP (missing feature)             50
  SKIP (run-time condition)          54

Testify:

Summary:

  GNU Octave Version: 4.4.1 (hg id: 1f46d371968c + patches)
  Tests run on angharad.local (macOS) at 19-Apr-2019 17:13:18
  Test execution time: 00:10:33

  PASS                            18090
  FAIL                               34
  XFAIL (reported bug)              222
  SKIP (missing feature)             23
  SKIP (run-time condition)          60

  Failed tests:
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/miscellaneous/inputname.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isosurface.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/reducevolume.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/copyobj.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/hgsave.m

And why is it reporting more PASSes, too? 18090 vs 14933.

Files with failed tests

And doesn't that seem like too few files to have 34 failures?

apjanke commented 5 years ago

inputname:

Aw, crap. At least some of the BISTs are depending on the internal implementation details of test.

Here's some test code from inputname.m:

## Warning: heap big magic in the following tests!!!
## The test function builds a private context for each test, with only the
## specified values shared between them.  It does this using the following
## template:
##
##     function [<shared>] = testfn (<shared>)
##        <test>
##     endfunction
##
## To test inputname, I need a function context invoked with known parameter
## names.  So define a couple of shared parameters, et voila!, the test is
## trivial.

%!shared hello, worldly
%!assert (inputname (1), "hello")
%!assert (inputname (2), "worldly")
%!assert (inputname (3), "")

## Clear parameter list so that testfn is created with zero inputs/outputs
%!shared
%!assert (inputname (-1), "")
%!assert (inputname (1), "")

That seems to rely on the fact that Octave's existing test implements shared variables by passing them as parameters to a testing function call that it generates dynamically, and that that testing function (with shared variables passed as named parameters) is the immediately enclosing stack frame in which statements in the test block are executed. That's incompatible with the object-managed workspace approach that Testify is using.

apjanke commented 5 years ago

isocaps:

One of the "error" tests is getting the wrong error message.

Octave's:

>> test isocaps verbose
>>>>> /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
[...]
***** error <X must match the size of V>
 x = 1:2:24;
 fvc = isocaps (x, y, z, val, iso);
***** error <Y must match the size of V>
 y = -14:6:11;
 fvc = isocaps (x, y, z, val, iso);
***** error <Z must match the size of V>
[...]

Testify's:

*****  testify.internal.BistBlock: error (index=24)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/V must be a non-singleton 3-dimensional matrix/
Code:

 v3 = reshape (1:6*8, [6 1 8]);
 fvc = isocaps (v3, iso);
  -> success=1, msg=
*****  testify.internal.BistBlock: error (index=25)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/X must match the size of V/
Code:

 x = 1:2:24;
 fvc = isocaps (x, y, z, val, iso);
  -> success=1, msg=
*****  testify.internal.BistBlock: error (index=26)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Y must match the size of V/
Code:

 y = -14:6:11;
 fvc = isocaps (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isocaps: X must match thesize of V'
PASSES 22 out of 23 tests

Now, why would that happen? Is it because we're persisting our shared variables differently?

apjanke commented 5 years ago

getframe:

This failure is intermittent. Seems to be related to test ordering. When I get a failure, it complains about something not being implemented for the gnuplot toolkit. Normally, I'm running with the qt toolkit; looks like the toolkit got switched during test execution somewhere?

Yeah: In 5.1.0, after I do a full __run_test_suite2__, my graphics_toolkit is left as gnuplot:

>> have_window_system, graphics_toolkit
ans = 1
ans = gnuplot
>> test2 getframe
PASSES 2 out of 2 tests
Skipped 2 tests due to run-time conditions
>>

But in 4.4.1, the tests aren't properly skipped:

>> version
ans = 4.4.1
>> have_window_system, graphics_toolkit
ans = 1
ans = gnuplot
>> test2 getframe
PASSES 0 out of 1 test
>> test2 getframe -verbose

>>>>> /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
[...]
*****  testify.internal.BistBlock: testif (index=5)
  is_test=1  is_xtest=0
  feature_line=' HAVE_QT_OFFSCREEN; have_window_system ()'
  feature='{QT_OFFSCREEN}'
  runtime_feature_test=' have_window_system ()'
Code:

 hf = figure ("visible", "off");
 unwind_protect
   pos = get (hf, "position");
   assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
  -> success=0, msg=test failed: raised error: getframe: not implemented for gnuplot graphics toolkit
PASSES 0 out of 1 test

Back on 5.1.0, it's showing a different testif condition for that same block:

*****  testify.internal.BistBlock: testif (index=5)
  is_test=1  is_xtest=0
  feature_line=' HAVE_QT_OFFSCREEN; have_window_system () && strcmp ("qt", graphics_toolkit ())'
  feature='{QT_OFFSCREEN}'
  runtime_feature_test=' have_window_system () && strcmp ("qt", graphics_toolkit ())'
Code:

 hf = figure ("visible", "off");
 unwind_protect
   pos = get (hf, "position");
   assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
  -> success=1, msg=----- skipped test (runtime test)

Ah, yes: the test condition actually did change between 4.4.1 and 5.1.0. On 4.4.1, plain test is failing, too:

>> test getframe
***** testif HAVE_QT_OFFSCREEN; have_window_system ()
 hf = figure ("visible", "off");
 unwind_protect
   pos = get (hf, "position");
   assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
!!!!! test failed
getframe: not implemented for gnuplot graphics toolkit
>>
apjanke commented 5 years ago

isosurface:

Similar test failure to isocaps, with an incorrect size-difference error message:


>> test2 isosurface -verbose
[...]
*****  testify.internal.BistBlock: error (index=25)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Y must match the size of V/
Code:

 y = -14:6:11;
 fvc = isosurface (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
PASSES 20 out of 21 tests
>>
apjanke commented 5 years ago

reducevolume:

Yet another problem with incorrect error messages for input size comparisons:

>> test reducevolume
PASSES 22 out of 22 tests
>> test2 reducevolume -no-fail-fast
----- Failure details: block 19 -----
*****  test block 19: error /X must match the size of data V/
  is_test=1 is_xtest=0
Code:
 x = 1:2:24;
 [nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /X must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 20 -----
*****  test block 20: error /Y must match the size of data V/
  is_test=1 is_xtest=0
Code:
 y = -14:6:11;
 [nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 21 -----
*****  test block 21: error /Z must match the size of data V/
  is_test=1 is_xtest=0
Code:
 z = linspace (16, 18, 5);
 [nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Z must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 22 -----
*****  test block 22: error /reduction value R is too high/
  is_test=1 is_xtest=0
Code:
 [nv] = reducevolume (v, 5)
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /reduction value R is too high/, but got 'reducevolume: data must be a non-singleton 3-dimensional matrix'
----- Failure details: block 23 -----
*****  test block 23: error /reduction value R is too high/
  is_test=1 is_xtest=0
Code:
 [nv] = reducevolume (v, [4 7 2])
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /reduction value R is too high/, but got 'reducevolume: data must be a non-singleton 3-dimensional matrix'
PASSES 17 out of 22 tests
>>
apjanke commented 5 years ago

copyobj and hgsave are failing under regular Octave test too. Marking them off.

apjanke commented 5 years ago

Results running under CLI vanilla Homebrewed Octave 5.1.0:


Summary:

  GNU Octave Version: 5.1.0 (hg id: d05d6eebde10)
  Tests run on angharad.local (macOS) at 19-Apr-2019 20:02:45
  Test execution time: 00:12:21

  PASS                            18810
  FAIL                               35
  XFAIL (reported bug)              247
  SKIP (missing feature)            176
  SKIP (run-time condition)          78

  Failed tests:
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/etc/tests/libinterp/corefcn/cellfun.cc-tst
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/etc/tests/libinterp/corefcn/mappers.cc-tst
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/general/logspace.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/miscellaneous/inputname.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/miscellaneous/tar.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/isocaps.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/isosurface.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/reducevolume.m

Those failures are more what I'm used to.

Full test results: testsuite-angharad.local-macOS-20190419_194818.zip

apjanke commented 5 years ago

I manually extracted all the failures from the progress output, and it matches the summary results... except all the M-file tests are being run twice!

  <Octave/tests>/libinterp/corefcn/cellfun.cc-tst ............. PASS    122/130 
                                                                FAIL      4
  <Octave/tests>/libinterp/corefcn/mappers.cc-tst ............. PASS    414/421 
                                                                FAIL      1
  <Octave/m>/general/logspace.m ............................... PASS     14/16  
                                                                FAIL      2
  <Octave/m>/miscellaneous/inputname.m ........................ PASS      9/11  
                                                                FAIL      2
  <Octave/m>/miscellaneous/tar.m .............................. PASS      5/6   
                                                                FAIL      1
  <Octave/m>/plot/draw/isocaps.m .............................. PASS     27/29  
                                                                FAIL      2
  <Octave/m>/plot/draw/isosurface.m ........................... PASS     26/30  
                                                                FAIL      3
  <Octave/m>/plot/draw/reducevolume.m ......................... PASS     17/22  
                                                                FAIL      5
  <Octave/m>/general/logspace.m ............................... PASS     14/16  
                                                                FAIL      2
  <Octave/m>/miscellaneous/inputname.m ........................ PASS      9/11  
                                                                FAIL      2
  <Octave/m>/miscellaneous/tar.m .............................. PASS      5/6   
                                                                FAIL      1
  <Octave/m>/plot/draw/isosurface.m ........................... PASS     26/30  
                                                                FAIL      3
  <Octave/m>/plot/draw/reducevolume.m ......................... PASS     17/22  
                                                                FAIL      5
apjanke commented 5 years ago

After fixing #78, I'm down to this:

Summary:

  GNU Octave Version: 4.4.1 (hg id: 1f46d371968c + patches)
  Tests run on angharad.local (macOS) at 19-Apr-2019 20:41:12
  Test execution time: 00:05:29

  PASS                            11260
  FAIL                               18
  XFAIL (reported bug)              157
  SKIP (missing feature)             14
  SKIP (run-time condition)          51

  Failed tests:
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/miscellaneous/inputname.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isosurface.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/reducevolume.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/copyobj.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/hgsave.m

Better. Now I think I'm down to legit individual test failures.