search

gnu-octave / octave-doctest

Doctests for Octave/Matlab
https://gnu-octave.github.io/packages/doctest/
BSD 3-Clause "New" or "Revised" License
16 stars 4 forks source link

string returns lead to Self-tests fail on Matlab #231

Open skycaptain opened 5 years ago

skycaptain commented 5 years ago

I'm using Matlab 2019b on MacOS. Running doctest doctest fails with:

>> doctest doctest
Doctest v0.7.0+: this is Free Software without warranty, see source.

doctest ................................................ FAIL    1/14  

   >> datestr(now, 'yyyy-mm-dd')

      expected:          2...

      got     : 
ans =

    '2019-09-30'

Matlab prints strings with apostrophs, Octave doesn't.

A way, which works with both Octave and Matlab is to use disp: disp(datestr(now, 'yyyy-mm-dd')).

Also, - and I hope this is not just a case of Just on my machine - most of the tests running make matlab_test either fail or are not found. Some but not all are releated to this issue:

>>> env MATLAB=$MATLAB_ROOT/bin/matlab make matlab_test
cd "tmp/doctest-matlab-0.7.0+"; /Applications/MATLAB_R2019b.app/bin/matlab -nojvm -nodisplay -nosplash -r "ver(), success = doctest({'doctest', 'test/', 'test/examples/'}); exit(~success);"

                                         < M A T L A B (R) >
                               Copyright 1984-2019 The MathWorks, Inc.
                               R2019b (9.7.0.1190202) 64-bit (maci64)
                                           August 21, 2019

For online documentation, see https://www.mathworks.com/support
For product information, visit www.mathworks.com.

EDIT: SKIPPING VER OUTPUT 

Doctest v0.7.0+: this is Free Software without warranty, see source.

doctest ................................................ PASS   14/14  
test/
  test_class ........................................... FAIL    1/2   

   >> class(test_class)

      expected:   ans = test_class

      got     : 
ans =

    'test_class'

  test_class.test_class ................................ FAIL    1/2   

   >> class(test_class)

      expected:   ans = test_class

      got     : 
ans =

    'test_class'

  test_class.test_method ............................... FAIL    1/1   

   >> m = test_class; test_method(m)

      expected:   ans = Default Name is 42 years old.

      got     : 
ans =

    'Default Name is 42 years old.'

  test_classdef ........................................ PASS    2/2   
  test_classdef.amethod ................................ PASS    1/1   
  test_classdef.disp ................................... PASS    1/1   
  test_classdef.test_classdef .......................... PASS    1/1   
  test_shadow .......................................... FAIL    1/1   

   >> 6 + 7

      expected:   ans = 13

    Folders named test_shadow

       test/test_shadow

      got     : 
ans =

    13

  test_shadow.test_shadow .............................. PASS    1/1   
  bist.m ............................................... NO TESTS
  classdef_infile.m .................................... PASS    2/2   
  examples/
    greet.m ............................................ FAIL    1/1   

   >> greet World

      expected:  
  Hello, World!

      got     : 
ans =

    'Hello, World!'

  private/
    test_in_private_dir.m .............................. NO TESTS
  test_angle_brackets.m ................................ FAIL    2/3   

   >> s = test_angle_brackets()    % doctest: +XFAIL_IF(oct38)

      expected:   s = I <3 U

      got     : 
s =

    'I <3 U'

   >> s = '<p>I heart you</p>'     % doctest: +XFAIL_IF(oct38)

      expected:   s = <p>I heart you</p>

      got     : 
s =

    '<p>I heart you</p>'

  test_ans.m ........................................... PASS    5/5   
  test_ans.texinfo ..................................... NO TESTS
  test_blank_match.m ................................... PASS    1/1   
  test_clear.m ......................................... FAIL    3/11  

   >> a

      expected:   ??? ...ndefined ...

      got     : ??? Unrecognized function or variable 'a'.

   >> a

      expected:   ??? ...ndefined ...

      got     : ??? Unrecognized function or variable 'a'.

   >> a

      expected:   ??? ...ndefined ...

      got     : ??? Unrecognized function or variable 'a'.

  test_clear_all_first.m ............................... FAIL    1/3   

   >> a

      expected:   ??? ...ndefined ...

      got     : ??? Unrecognized function or variable 'a'.

  test_clear_isoctave.m ................................ FAIL    2/6   

   >> a

      expected:   ??? ...ndefined ...

      got     : ??? Unrecognized function or variable 'a'.

   >> a

      expected:   ??? ...ndefined ...

      got     : ??? Unrecognized function or variable 'a'.

  test_comments.texinfo ................................ NO TESTS
  test_comments_with_directives.m ...................... PASS    3/3   
  test_compare_backspace.m ............................. FAIL    1/1   

   >> if (exist('OCTAVE_VERSION'))
      eval('sprintf("Hi, no question mark here?\x08 goodbye")')
    else
      sprintf('Hi, no question mark here?\x08 goodbye')
    end

      expected:  
  ans =

  Hi, no question mark here goodbye

      got     : 
ans =

    'Hi, no question mark here goodbye'

  test_compare_hyperlinks.m ............................ PASS    1/1   
  test_diary_style.texinfo ............................. NO TESTS
  test_diary_style_mixed.texinfo ....................... NO TESTS
  test_dir/
    test1.m ............................................ PASS    1/1   
    test_subdir/
      test2.m .......................................... PASS    1/1   
  test_ellipsis.m ...................................... FAIL    1/4   

   >> '...'   % doctest: -ELLIPSIS

      expected:  
  ...

      got     : 
ans =

    '...'

  test_ellipsis_match_empty.m .......................... PASS    5/5   
  test_ellipsis_match_whitespace.m ..................... PASS    9/9   
  test_format.m ........................................ PASS    5/5   
  test_long_rows.m ..................................... PASS    1/1   
  test_long_rows.texinfo ............................... NO TESTS
  test_methods_in_subdir/
    logical ............................................ NO TESTS
    logical.abs ........................................ NO TESTS
    logical.accumarray ................................. NO TESTS
    logical.all ........................................ NO TESTS
    logical.amd ........................................ NO TESTS
    logical.and ........................................ NO TESTS
    logical.any ........................................ NO TESTS
    logical.bitand ..................................... NO TESTS
    logical.bitcmp ..................................... NO TESTS
    logical.bitget ..................................... NO TESTS
    logical.bitor ...................................... NO TESTS
    logical.bitset ..................................... NO TESTS
    logical.bitshift ................................... NO TESTS
    logical.bitxor ..................................... NO TESTS
    logical.bsxfun ..................................... NO TESTS
    logical.ceil ....................................... NO TESTS
    logical.circshift .................................. NO TESTS
    logical.complex .................................... NO TESTS
    logical.conv2 ...................................... NO TESTS
    logical.ctranspose ................................. NO TESTS
    logical.cummax ..................................... NO TESTS
    logical.cummin ..................................... NO TESTS
    logical.cumprod .................................... NO TESTS
    logical.cumsum ..................................... NO TESTS
    logical.diag ....................................... NO TESTS
    logical.diff ....................................... NO TESTS
    logical.display .................................... NO TESTS
    logical.dmperm ..................................... NO TESTS
    logical.eq ......................................... NO TESTS
    logical.fft ........................................ NO TESTS
    logical.fftn ....................................... NO TESTS
    logical.filter ..................................... NO TESTS
    logical.find ....................................... NO TESTS
    logical.fix ........................................ NO TESTS
    logical.flip ....................................... NO TESTS
    logical.floor ...................................... NO TESTS
    logical.full ....................................... NO TESTS
    logical.ge ......................................... NO TESTS
    logical.gt ......................................... NO TESTS
    logical.ifft ....................................... NO TESTS
    logical.ifftn ...................................... NO TESTS
    logical.imag ....................................... NO TESTS
    logical.iscolumn ................................... NO TESTS
    logical.isempty .................................... NO TESTS
    logical.isequal .................................... NO TESTS
    logical.isequaln ................................... NO TESTS
    logical.isequalwithequalnans ....................... NO TESTS
    logical.isfinite ................................... NO TESTS
    logical.isinf ...................................... NO TESTS
    logical.ismatrix ................................... NO TESTS
    logical.isnan ...................................... NO TESTS
    logical.isreal ..................................... NO TESTS
    logical.isrow ...................................... NO TESTS
    logical.isscalar ................................... NO TESTS
    logical.issorted ................................... NO TESTS
    logical.issortedrows ............................... NO TESTS
    logical.issparse ................................... NO TESTS
    logical.isvector ................................... NO TESTS
    logical.ldivide .................................... NO TESTS
    logical.le ......................................... NO TESTS
    logical.length ..................................... NO TESTS
    logical.linsolve ................................... NO TESTS
    logical.lt ......................................... NO TESTS
    logical.max ........................................ NO TESTS
    logical.maxk ....................................... NO TESTS
    logical.min ........................................ NO TESTS
    logical.mink ....................................... NO TESTS
    logical.minus ...................................... NO TESTS
    logical.mldivide ................................... NO TESTS
    logical.mod ........................................ NO TESTS
    logical.mrdivide ................................... NO TESTS
    logical.mtimes ..................................... NO TESTS
    logical.mynewmethod ................................ FAIL    2/3   

   >> mynewmethod(true)

      expected:     ans = 0

      got     : 
ans =

  <a href="matlab:helpPopup logical" style="font-weight:bold">logical</a>

   0

   >> islogical(ans)

      expected:     ans = 1

      got     : 
ans =

  <a href="matlab:helpPopup logical" style="font-weight:bold">logical</a>

   1

    logical.ndims ...................................... NO TESTS
    logical.ne ......................................... NO TESTS
    logical.nnz ........................................ NO TESTS
    logical.nonzeros ................................... NO TESTS
    logical.norm ....................................... NO TESTS
    logical.not ........................................ NO TESTS
    logical.numel ...................................... NO TESTS
    logical.nzmax ...................................... NO TESTS
    logical.or ......................................... NO TESTS
    logical.permute .................................... NO TESTS
    logical.plus ....................................... NO TESTS
    logical.power ...................................... NO TESTS
    logical.prod ....................................... NO TESTS
    logical.rdivide .................................... NO TESTS
    logical.real ....................................... NO TESTS
    logical.rem ........................................ NO TESTS
    logical.repelem .................................... NO TESTS
    logical.repmat ..................................... NO TESTS
    logical.reshape .................................... NO TESTS
    logical.round ...................................... FAIL    1/1   

   >> round(pi, 2)

      expected:          3.14

    % Round the equatorial radius of the Earth, 6378137 meters,
    % to the nearest kilometer.
    round(6378137, -3)
         6378000

    % Round to 3 significant digits
    format shortg;
    round([pi, 6378137], 3, 'significant')
         3.14     6.38e+06

    If you only need to display a rounded version of X,
    consider using fprintf or num2str:

    fprintf('%.3f\n', 12.3456)
         12.346 
    fprintf('%.3e\n', 12.3456)
         1.235e+01

   See also FLOOR, CEIL, FPRINTF.

      got     : 
ans =

    3.1400

    logical.sign ....................................... NO TESTS
    logical.size ....................................... NO TESTS
    logical.sort ....................................... NO TESTS
    logical.sortrowsc .................................. NO TESTS
    logical.sparse ..................................... NO TESTS
    logical.sum ........................................ NO TESTS
    logical.superiorfloat .............................. NO TESTS
    logical.symrcm ..................................... NO TESTS
    logical.times ...................................... NO TESTS
    logical.transpose .................................. NO TESTS
    logical.tril ....................................... NO TESTS
    logical.triu ....................................... NO TESTS
    logical.uminus ..................................... NO TESTS
    logical.uplus ...................................... NO TESTS
    logical.xor ........................................ NO TESTS
  test_multi_directives.m .............................. PASS    5/5   
  test_multi_result.texinfo ............................ NO TESTS
  test_multi_return.texinfo ............................ NO TESTS
  test_no_docs.m ....................................... NO TESTS
  test_shadow .......................................... FAIL    1/1   

   >> 6 + 7

      expected:   ans = 13

    Folders named test_shadow

       test/test_shadow

      got     : 
ans =

    13

  test_shadow.test_shadow .............................. PASS    1/1   
  test_shadow.m ........................................ PASS    1/1   
  test_skip.m .......................................... PASS    3/3   
  test_skip_comments.texinfo ........................... NO TESTS
  test_skip_if.m ....................................... PASS    4/4   
  test_skip_if_multiple.m .............................. PASS    5/5   
  test_skip_malformed.texinfo .......................... NO TESTS
  test_skip_only_one.m ................................. NO TESTS
  test_skip_unless.m ................................... PASS    4/4   
  test_skip_wo_output.texinfo .......................... NO TESTS
  test_skip_wo_output_diary.m .......................... PASS    2/2   
  test_skip_wo_output_diary.texinfo .................... NO TESTS
  test_var.texinfo ..................................... NO TESTS
  test_warning.m ....................................... PASS    1/1   
  test_whitespace.m .................................... PASS    8/8   
  test_windows_eol.texinfo ............................. NO TESTS
  test_xfail.m ......................................... PASS    3/3   
  test_xfail.texinfo ................................... NO TESTS
  test_xfail_if.m ...................................... PASS    6/6   
  test_xfail_if_code.m ................................. PASS    5/5   
  test_xfail_if_code.texinfo ........................... NO TESTS
  test_xfail_if_multiple.m ............................. PASS    7/7   
  test_xfail_unless.m .................................. PASS    6/6   
  test_xskip_if_code.texinfo ........................... NO TESTS
test/examples/
  greet.m .............................................. FAIL    1/1   

   >> greet World

      expected:  
  Hello, World!

      got     : 
ans =

    'Hello, World!'

Summary:

   FAIL   20/156 

158/173 targets passed, 126 without tests.

make: *** [matlab_test] Error 1
cbm755 commented 5 years ago

Maybe there is a change in 2019b about printing strings? On old Matlab,

>> a = 'hello'
a = 
hello

Is that perhaps not the case in 2019b? Is that documented in their release notes?

cbm755 commented 5 years ago

Maybe even earlier: somewhere between 2015 and 2018...

skycaptain commented 5 years ago

Must be earlier. It's the same with R2019a.

mtmiller commented 5 years ago

Maybe even earlier: somewhere between 2015 and 2018...

My guess is R2016b, when the string array object was introduced. Display needs to show the quote character to differentiate between char arrays and strings.

catch22 commented 5 years ago

As a lower bound, I'm pretty sure I was running the tests with 2017a at some point (on macOS).

skycaptain commented 5 years ago

So how can we fix this? Imho, rather than fixing this in code, the tests should be fixed to print the same results in Matlab and Octave, like with disp.

What do you think?

cbm755 commented 5 years ago

Actually I like this change. Maybe we should propose it for Octave too.

Anyway, as @skycaptain suggests, we should use disp where appropriate in our test set. For example: README.md and test/examples/ could call disp instead of returning a string. @skycaptain do you want to do this?

skycaptain commented 5 years ago

OK. I'll do a PR.

cbm755 commented 5 years ago

I filed https://savannah.gnu.org/bugs/index.php?56973 upstream

cbm755 commented 5 years ago

Actually, why does test_angle_brackets pass? Doesn't it need a disp fix too?

skycaptain commented 5 years ago

I think you meant doesn't pass, right? I left this out from the PR because of the assignment to s, but that's arguable, because that's not what this test is about.

cbm755 commented 5 years ago

Maybe I did.

I think it would be ok to change it to disp(test_angle_brackets())

I'll leave this open until we do that or similar. Are there any other failures on 2019b other than this and #234?

skycaptain commented 5 years ago
cbm755 commented 5 years ago

Heh logical.round isn't our file: this is an example of #87 and how we deal with superclasses.

I cannot reproduce any of these on 2018b.

cbm755 commented 5 years ago

re: shadow: some subtle precedence rules changed in 2019b so the comments in test_shadow.m maybe are not true? Not sure if this would lead to a test failure or not... I'll dig a little into that and maybe add a BIST about what I think should be happening here (see bist.m)