feature request: progress indicator for long running tests

[mtmiller]

Whether it's test or runtests, whatever the porcelain function will be that runs a sequence of tests.

Currently, if one particular file has hundreds of test cases or test cases that take time, the user sees

  foo.m .......................................................

followed by some unknown time span, finally followed by

  foo.m ....................................................... PASS    100/100

Would be nice to have an ascii or unicode animation showing that tests are running.

Inspired by Octave #43316.

[apjanke]

Sorry for the slow response on this. I didn’t have “Watch” turned on for this repo, so I wasn’t getting notifications for bug report submissions.

Sounds good. I think we need two things in order to do this:

• A good indicator of interactive vs batch mode for test execution
• A sort of event-handling system so the lower-level test-running code can pass notifications of progress steps elapsing up to the higher-level code that's handling output

For the batch vs interactive indicator, we may be able to test whether Octave is connected to a terminal to set the default value. But that won't work in the GUI Octave. And I'd like to have a function that you can run interactively that generates a report file, too. Maybe we could have interactive default to whether you're attached to a terminal, and have __run_test_suite__ or a similar top-level function be a batch-mode invoker that overrides that and forces it to be batch mode.

[apjanke]

Looking at Octave #43316, there might be two separate use cases here. For keeping an automated process from timing out, you probably want a different sort of progress indicator than an interactive spinner/throbber animation: the animation uses backspaces or terminal cursor movement commands, which are inappropriate for inclusion into a log file. Automated processes should receive append-only progress indicators that play well with being saved to a file.

[apjanke]

Do you also care about preventing timeouts during the build/install stage of package installations? Because those are done with single calls to make and configure, and Octave is single-threaded, that would probably need to be done with a wrapper for the make/configure calls that used something like bash coprocesses to emit the progress indicators while make is off doing its thing.

[mtmiller]

Thanks for taking an interest! This was just an off the cuff idea taken from the bug I referenced.

A sort of event-handling system so the lower-level test-running code can pass notifications of progress steps elapsing up to the higher-level code that's handling output

Right, as a first step the easiest would be a pulse after every individual test block. After that this could get more complicated into maybe double parsing so the total number of tests is known ahead of time, maybe other enhancements.

[mtmiller]

Do you also care about preventing timeouts during the build/install stage of package installations?

I personally don't, but I could see the case for that being an enhancement to pkg install too. In automated processes I would typically use pkg install -verbose, so no improvement needed there. Only users doing pkg install foo and wondering what's taking so long.

[apjanke]

Work is in progress on the https://github.com/apjanke/octave-testify/tree/progress-indicators branch. I’ve written a basic throbber progress indicator class, but not integrated it into any of the test scripts yet.

To test it, do:

testify.internal.Throbber.demo_all
% or:
thr = testify.internal.Throbber.get_throbber ("A");
thr.demo;
thr = testify.internal.Throbber.get_throbber ("clocks");
thr.demo;

[mtmiller]

Very cool so far.

Some minor issues with a couple of the styles in my terminal:

[apjanke]

Ah, crap. That cards thing is an issue with the number-of-columns used by the terminal for display. This is a hard issue to fix, because it doesn't depend on the properties of the Unicode characters; it depends on rendering decisions in both the font you're using and your particular terminal emulator, and there's no good way for a client program like Octave to detect that stuff.

How are "smileys", "dominoes", and "clocks" working for you there?

That's Terminator, right? What version and what OS? And what font(s) are you using?

And "hmmm" is broken everywhere; I dunno why. Sorry.

[mtmiller]

Yeah, Terminator on Debian with GNOME, the font is Noto Mono.

Only "cards" and "dominoes" go backwards and overwrite the "Doing stuff", all the emoji work fine.

[apjanke]

Okay. It sounds like Terminator + Noto Mono are combining to make Terminator think that cards and dominoes are half-width (1 column) instead of double-width (2 columns). So when I backspace 2 times to overwrite the card character, it effectively does an extra backspace.

This will be really hard to fix, I think, for the above reasons. Do you mind if we just let this one slide, at least for the time being? It's not like it's critical functionality.

[mtmiller]

Yeah absolutely put that off until much later.

[apjanke]

We'll probably need isatty support, since that doesn't seem to be present in Octave. That'll mean adding oct-files to the package.

[mtmiller]

We'll probably need isatty support

Should we get that into Octave 6? Seems easy enough.

[apjanke]

Should we get that into Octave 6?

That would be nice. But I think we should do it here first. That way we can get it working sooner, and have Testify be compatible with Octave 5.x and 4.x, too, which I think is of interest. That would be a good way to test isatty before submitting it for inclusion in core Octave, too.

[mtmiller]

Easy and fair enough. doctest used to define its own evalc function before that showed up in Octave.