Open apjanke opened 5 years ago
inputname
:
Aw, crap. At least some of the BISTs are depending on the internal implementation details of test
.
Here's some test code from inputname.m
:
## Warning: heap big magic in the following tests!!!
## The test function builds a private context for each test, with only the
## specified values shared between them. It does this using the following
## template:
##
## function [<shared>] = testfn (<shared>)
## <test>
## endfunction
##
## To test inputname, I need a function context invoked with known parameter
## names. So define a couple of shared parameters, et voila!, the test is
## trivial.
%!shared hello, worldly
%!assert (inputname (1), "hello")
%!assert (inputname (2), "worldly")
%!assert (inputname (3), "")
## Clear parameter list so that testfn is created with zero inputs/outputs
%!shared
%!assert (inputname (-1), "")
%!assert (inputname (1), "")
That seems to rely on the fact that Octave's existing test
implements shared variables by passing them as parameters to a testing function call that it generates dynamically, and that that testing function (with shared variables passed as named parameters) is the immediately enclosing stack frame in which statements in the test block are executed. That's incompatible with the object-managed workspace approach that Testify is using.
isocaps
:
One of the "error" tests is getting the wrong error message.
Octave's:
>> test isocaps verbose
>>>>> /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
[...]
***** error <X must match the size of V>
x = 1:2:24;
fvc = isocaps (x, y, z, val, iso);
***** error <Y must match the size of V>
y = -14:6:11;
fvc = isocaps (x, y, z, val, iso);
***** error <Z must match the size of V>
[...]
Testify's:
***** testify.internal.BistBlock: error (index=24)
is_test=1 is_xtest=0
error_id=''
pattern=/V must be a non-singleton 3-dimensional matrix/
Code:
v3 = reshape (1:6*8, [6 1 8]);
fvc = isocaps (v3, iso);
-> success=1, msg=
***** testify.internal.BistBlock: error (index=25)
is_test=1 is_xtest=0
error_id=''
pattern=/X must match the size of V/
Code:
x = 1:2:24;
fvc = isocaps (x, y, z, val, iso);
-> success=1, msg=
***** testify.internal.BistBlock: error (index=26)
is_test=1 is_xtest=0
error_id=''
pattern=/Y must match the size of V/
Code:
y = -14:6:11;
fvc = isocaps (x, y, z, val, iso);
-> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isocaps: X must match thesize of V'
PASSES 22 out of 23 tests
Now, why would that happen? Is it because we're persisting our shared variables differently?
getframe
:
This failure is intermittent. Seems to be related to test ordering. When I get a failure, it complains about something not being implemented for the gnuplot toolkit. Normally, I'm running with the qt toolkit; looks like the toolkit got switched during test execution somewhere?
Yeah: In 5.1.0, after I do a full __run_test_suite2__
, my graphics_toolkit is left as gnuplot:
>> have_window_system, graphics_toolkit
ans = 1
ans = gnuplot
>> test2 getframe
PASSES 2 out of 2 tests
Skipped 2 tests due to run-time conditions
>>
But in 4.4.1, the tests aren't properly skipped:
>> version
ans = 4.4.1
>> have_window_system, graphics_toolkit
ans = 1
ans = gnuplot
>> test2 getframe
PASSES 0 out of 1 test
>> test2 getframe -verbose
>>>>> /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
[...]
***** testify.internal.BistBlock: testif (index=5)
is_test=1 is_xtest=0
feature_line=' HAVE_QT_OFFSCREEN; have_window_system ()'
feature='{QT_OFFSCREEN}'
runtime_feature_test=' have_window_system ()'
Code:
hf = figure ("visible", "off");
unwind_protect
pos = get (hf, "position");
assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
unwind_protect_cleanup
close (hf);
end_unwind_protect
-> success=0, msg=test failed: raised error: getframe: not implemented for gnuplot graphics toolkit
PASSES 0 out of 1 test
Back on 5.1.0, it's showing a different testif condition for that same block:
***** testify.internal.BistBlock: testif (index=5)
is_test=1 is_xtest=0
feature_line=' HAVE_QT_OFFSCREEN; have_window_system () && strcmp ("qt", graphics_toolkit ())'
feature='{QT_OFFSCREEN}'
runtime_feature_test=' have_window_system () && strcmp ("qt", graphics_toolkit ())'
Code:
hf = figure ("visible", "off");
unwind_protect
pos = get (hf, "position");
assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
unwind_protect_cleanup
close (hf);
end_unwind_protect
-> success=1, msg=----- skipped test (runtime test)
Ah, yes: the test condition actually did change between 4.4.1 and 5.1.0. On 4.4.1, plain test
is failing, too:
>> test getframe
***** testif HAVE_QT_OFFSCREEN; have_window_system ()
hf = figure ("visible", "off");
unwind_protect
pos = get (hf, "position");
assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
unwind_protect_cleanup
close (hf);
end_unwind_protect
!!!!! test failed
getframe: not implemented for gnuplot graphics toolkit
>>
isosurface
:
Similar test failure to isocaps
, with an incorrect size-difference error message:
>> test2 isosurface -verbose
[...]
***** testify.internal.BistBlock: error (index=25)
is_test=1 is_xtest=0
error_id=''
pattern=/Y must match the size of V/
Code:
y = -14:6:11;
fvc = isosurface (x, y, z, val, iso);
-> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
PASSES 20 out of 21 tests
>>
reducevolume
:
Yet another problem with incorrect error messages for input size comparisons:
>> test reducevolume
PASSES 22 out of 22 tests
>> test2 reducevolume -no-fail-fast
----- Failure details: block 19 -----
***** test block 19: error /X must match the size of data V/
is_test=1 is_xtest=0
Code:
x = 1:2:24;
[nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /X must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 20 -----
***** test block 20: error /Y must match the size of data V/
is_test=1 is_xtest=0
Code:
y = -14:6:11;
[nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 21 -----
***** test block 21: error /Z must match the size of data V/
is_test=1 is_xtest=0
Code:
z = linspace (16, 18, 5);
[nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Z must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 22 -----
***** test block 22: error /reduction value R is too high/
is_test=1 is_xtest=0
Code:
[nv] = reducevolume (v, 5)
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /reduction value R is too high/, but got 'reducevolume: data must be a non-singleton 3-dimensional matrix'
----- Failure details: block 23 -----
***** test block 23: error /reduction value R is too high/
is_test=1 is_xtest=0
Code:
[nv] = reducevolume (v, [4 7 2])
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /reduction value R is too high/, but got 'reducevolume: data must be a non-singleton 3-dimensional matrix'
PASSES 17 out of 22 tests
>>
copyobj
and hgsave
are failing under regular Octave test
too. Marking them off.
Results running under CLI vanilla Homebrewed Octave 5.1.0:
Summary:
GNU Octave Version: 5.1.0 (hg id: d05d6eebde10)
Tests run on angharad.local (macOS) at 19-Apr-2019 20:02:45
Test execution time: 00:12:21
PASS 18810
FAIL 35
XFAIL (reported bug) 247
SKIP (missing feature) 176
SKIP (run-time condition) 78
Failed tests:
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/etc/tests/libinterp/corefcn/cellfun.cc-tst
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/etc/tests/libinterp/corefcn/mappers.cc-tst
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/general/logspace.m
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/miscellaneous/inputname.m
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/miscellaneous/tar.m
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/isocaps.m
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/isosurface.m
/usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/reducevolume.m
Those failures are more what I'm used to.
Full test results: testsuite-angharad.local-macOS-20190419_194818.zip
I manually extracted all the failures from the progress output, and it matches the summary results... except all the M-file tests are being run twice!
<Octave/tests>/libinterp/corefcn/cellfun.cc-tst ............. PASS 122/130
FAIL 4
<Octave/tests>/libinterp/corefcn/mappers.cc-tst ............. PASS 414/421
FAIL 1
<Octave/m>/general/logspace.m ............................... PASS 14/16
FAIL 2
<Octave/m>/miscellaneous/inputname.m ........................ PASS 9/11
FAIL 2
<Octave/m>/miscellaneous/tar.m .............................. PASS 5/6
FAIL 1
<Octave/m>/plot/draw/isocaps.m .............................. PASS 27/29
FAIL 2
<Octave/m>/plot/draw/isosurface.m ........................... PASS 26/30
FAIL 3
<Octave/m>/plot/draw/reducevolume.m ......................... PASS 17/22
FAIL 5
<Octave/m>/general/logspace.m ............................... PASS 14/16
FAIL 2
<Octave/m>/miscellaneous/inputname.m ........................ PASS 9/11
FAIL 2
<Octave/m>/miscellaneous/tar.m .............................. PASS 5/6
FAIL 1
<Octave/m>/plot/draw/isosurface.m ........................... PASS 26/30
FAIL 3
<Octave/m>/plot/draw/reducevolume.m ......................... PASS 17/22
FAIL 5
After fixing #78, I'm down to this:
Summary:
GNU Octave Version: 4.4.1 (hg id: 1f46d371968c + patches)
Tests run on angharad.local (macOS) at 19-Apr-2019 20:41:12
Test execution time: 00:05:29
PASS 11260
FAIL 18
XFAIL (reported bug) 157
SKIP (missing feature) 14
SKIP (run-time condition) 51
Failed tests:
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/miscellaneous/inputname.m
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isosurface.m
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/reducevolume.m
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/copyobj.m
/Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/hgsave.m
Better. Now I think I'm down to legit individual test failures.
Testify is failing more tests than core Octave is.
And why is it reporting more PASSes, too? 18090 vs 14933.
Files with failed tests
libinterp/corefcn/pr-output.cc-tst
getframe.m
inputname.m
isocaps.m
isosurface.m
reducevolume.m
copyobj.m
hgsave.m
And doesn't that seem like too few files to have 34 failures?