feature request: option to save workspace on test failure

[mtmiller]

Often tests that process random data fail on corner cases. I've found it very helpful in the past to hack a save command into __run_test_suite__ on test failures so I can reproduce the failure with the exact data that caused it.

[apjanke]

I think this is doable. Do you want a snapshot of the workspace from before the failed test/test block, after, or both? And would you prefer to have it saved to somewhere in-process like appdata, or saved to a file?

[mtmiller]

Good questions, not sure what the ideal outcome would be, but here are a couple use cases where I've saved test data before.

1. Simple case. A test fails reliably every time. If I want to inspect why, I can run test foo verbose and copy and paste the workspace dump to my command line. This would be a convenience, if the test fails, automatically stash the test local vars somewhere and tell me where they are so I can load them. Could be in a data file in tempdir or stashed in Octave memory somewhere.

2. Random test failure. A test fails one out of every 1000 runs. I run octave in a bash loop to run the test until it fails, or in an octave for loop until it fails. After it fails, I would want to stash the local variables so I can look at them later or share them with another Octave session or load them on a different system. In this use case a file on disk would be more useful.

[apjanke]

Gotcha.

Let's do the simple case 1 first and see how it works. 2 will take more work because that'll involve figuring out a decent path construction convention for storing the resulting workspace files (and maybe some test result info to go along with them).

[apjanke]

Both cases are done (in https://github.com/apjanke/octave-testify/commit/736fdd8408fe2142611f2427a5d9cf7380ee7f18), as long as you don't mind concentrating on a single test failure for case 2 (instead of saving multiple workspaces for multiple test failures within a single test run).

There are new -fail-fast and -save-workspace options to test2_refactor and runtests2_refactor. When -save-workspace is enabled, it will save the "before" and "after" workspace states for a failed test to a temp file, and print the path to that file for you to use.

The -fail-fast behavior causes the test run to abort at the first test failure, so you can actually make use of that saved workspace, before it gets overwritten by subsequent test runs.

>> runtests2_refactor -fail-fast -save-workspace
Processing files for directory .:

Processing files for directory /Users/janke/Library/Application Support/Octave.app/5.1.0/pkg/doctest-0.6.1:
[...]
  /Users/janke/local/repos/octave-testify/inst/test2.m ........
Saved test workspace is available in: /var/folders/_4/9mx5ryp52bb_z6drbcbrhwl40000gn/T/octave-testify-BistRunner/bist-run-2019-03-19_17-34-03/workspaces/test-workspaces.mat
 PASS     35/36
                                                                  FAIL    1

>> test2_refactor eigs -save-workspace verbose

Saved test workspace is available in: /var/folders/_4/9mx5ryp52bb_z6drbcbrhwl40000gn/T/octave-testify-BistRunner/bist-run-2019-03-19_17-34-36/workspaces/test-workspaces.mat
PASSES 2 out of 3 tests
>>

You do not need to pass -fail-fast to test2, because fail-fast is the default behavior at that level. You do need to explicitly pass it to runtests2 if you want to use -save-workspace there.

The saved workspace file is a .mat file that contains two variables:

• before - the state of the test workspace before the failed test block was run
• after - the state of the test workspace after the failed test block was run

There is a performance penalty for this, because BistRunner pre-emptively saves out the "before" workspace on every test run, because it doesn't know if it's going to fail or not. I'm thinking through whether we could stash that in memory instead (using COW to save space) without unduly affecting the behavior of the test code WRT object lifetimes of shared variables. I think it might be okay, because the shared-workspace mechanism already holds on to those references until the end of the test, due to how the wax-on/wax-off mechanism of BistWorkspace works.

What do you think?

[mtmiller]

I tried a simple failing test case

%!test
%! x = 1;
%! y = rand ();
%! assert (x + y < 1)

And I got before and after structs with no variables in them

>> w = load ('/…/test-workspaces.mat');
>> w.before
ans =

  scalar structure containing the fields:

>> w.after
ans =

  scalar structure containing the fields:

Assuming that can be fixed, this looks to me like a useful solution.

[apjanke]

Oh, I thought only "shared" variables were supposed to persist in the test workspaces.

We can handle this case too. Looks to me like that is one "block", and the variables x and y should only live for the duration of that one block, but previously declared shared variables persist over multiple blocks?

[mtmiller]

Right.

Ok, so I've converted the bad test to

%!shared x, y
%! x = 1;
%! y = rand ();
%!test
%! assert (x + y < 1)

The test fails as expected, now I have a before and after with x and y each, but all are set to [](0x0).

[apjanke]

Well that's not right. I'll look in to it.

[apjanke]

Fixed in https://github.com/apjanke/octave-testify/commit/d1a28be2a10d36712d356335d61785a9acf9a7e7 and https://github.com/apjanke/octave-testify/commit/6d92c8565f518a0b8cbde6aaf4e2cff5e493ee8d.

octave:2> type myfailingtest.m
myfailingtest.m is the user-defined function defined from: /Users/janke/local/repos/octave-testify/myfailingtest.m

function myfailingtest ()
  hello = 42;
endfunction

%!test
%! x = 1;
%! y = 3;
%! assert (x + y < 1)

octave:3> test2 myfailingtest.m -save-workspace

Saved test workspace is available in: /var/folders/_4/9mx5ryp52bb_z6drbcbrhwl40000gn/T/octave/testify/BistRunner/bist-run-2019-03-28_22-38-41/workspaces/test-workspaces.mat
PASSES 0 out of 1 test
octave:4> wkspc = load('/var/folders/_4/9mx5ryp52bb_z6drbcbrhwl40000gn/T/octave/testify/BistRunner/bist-run-2019-03-28_22-38-41/workspaces/test-workspaces.mat')
wkspc =
  scalar structure containing the fields:
    after =
      scalar structure containing the fields:
        x =  1
        y =  3
    before =
      scalar structure containing the fields:

octave:5> type myfailingtest1
error: type: 'myfailingtest1' undefined
octave:5> type myfailingtest2
myfailingtest2 is the user-defined function defined from: /Users/janke/local/repos/octave-testify/myfailingtest2.m

function myfailingtest2 ()
  hello = 42;
endfunction

%!shared x, y
%! fprintf ("myfailingtest2: In shared initialization code\n");
%! x = 1;
%! fprintf ("Initialized x = %f\n", x);
%! y = rand ();
%! fprintf ("Initialized y = %f\n", y);
%!test
%! z = x + y
%! assert (z < 1)

octave:6> test2 myfailingtest2.m -save-workspace
myfailingtest2: In shared initialization code
Initialized x = 1.000000
Initialized y = 0.622033
z =  1.622033461617844

Saved test workspace is available in: /var/folders/_4/9mx5ryp52bb_z6drbcbrhwl40000gn/T/octave/testify/BistRunner/bist-run-2019-03-28_22-39-12/workspaces/test-workspaces.mat
PASSES 0 out of 1 test
octave:7> wkspc = load('/var/folders/_4/9mx5ryp52bb_z6drbcbrhwl40000gn/T/octave/testify/BistRunner/bist-run-2019-03-28_22-39-12/workspaces/test-workspaces.mat')
wkspc =
  scalar structure containing the fields:
    after =
      scalar structure containing the fields:
        x =  1
        y =    6.220334616178436e-01
        z =  1.622033461617844
    before =
      scalar structure containing the fields:
        x =  1
        y =    6.220334616178436e-01

octave:8>