Wrong PASS count for testing netcdf package

[apjanke]

Under Octave.app 4.4.1-u1:

>> pkg install -forge netcdf
2019-10-21 01:23 octave-gui[6052] (FSEvents.framework) FSEventStreamFlushSync(): failed assertion '(SInt64)last_id > 0LL'

2019-10-21 01:23 octave-gui[6052] (FSEvents.framework) FSEventStreamFlushSync(): failed assertion '(SInt64)last_id > 0LL'

For information about changes from previous versions of the netcdf package, run 'news netcdf'.
>> pkg load netcdf
>> runtests2 -pkg netcdf
  /Users/janke/Library/Application Support/Octave.app/4.4.1/pkg/netcdf-1.0.12/test_netcdf.m
Using NetCDF library version "4.6.3 of May  8 2019 00:09:03 $"
run test_netcdf_constant..............   OK
run test_netcdf_create................   OK
run test_netcdf_low_level_interface...   OK
run test_netcdf_unlim.................   OK
run test_netcdf_datatypes.............   OK
run test_netcdf_scalar_variable.......   OK
run test_netcdf_attributes............   OK
run test_netcdf_high_level_interface..   OK
run test_netcdf_ncwriteschema.........   OK
run test_netcdf_ncwriteschema_unlim...   OK
run test_netcdf_ncwriteschema_chunking   OK
run test_netcdf_ncwriteschema_group...   OK
 PASS      1/1

Summary:

  GNU Octave Version: 4.4.1 (hg id: 1f46d371968c + patches)
  Tests run on eilonwy.local (macOS) at 21-Oct-2019 01:24:09
  Test execution time: 00:00:00

  PASS                                1
  FAIL                                0

>>

[apjanke]

Ah. Looks like this is a problem with the netcdf package, in that it defines its tests in a nonstandard way. It wrote its tests in a test_netcdf.m function that contains a single test block:

%!test
%! fprintf ("\n");
%! test_netcdf ();

Which calls its own custom test runner code:

function test_netcdf()

import_netcdf

tests = {'test_netcdf_constant',...
       [...]
         'test_netcdf_ncwriteschema_group'...
        };

[...]
for iindex=1:length(tests);
  dots = repmat('.',1,maxlen - length(tests{iindex}));
  fprintf('run %s%s ',tests{iindex},dots);
  try
    eval(tests{iindex});
    disp('  OK  ');
  catch
    disp(' FAIL ');
    disp(lasterr)
  end
end

So it's avoiding standard integration with Octave's normal test framework.

I'm going to call this an upstream problem or quirk of the netcdf package, and not a problem with Testify: Testify is behaving as expected, because from the Octave test definition perspective, netcdf contains a single test block.

Closing as upstream/works-as-designed.