Dune is mangling qtest/ounit output

tperami commented 4 years ago

I'm trying to enable tests via qtests in dune. When I do dune test the output is garbled because Qtest/Ounit expect it's output do be directly on the terminal but try to process it in some way

Expected Behavior

$ dune test
        qtest .test.inline-tests/run.ml-gen
Extraction : `mod.ml' Done.
         run alias runtest
random seed: 433741396
Ran: 1 tests in: 0.00 seconds.                                        
SUCCESS

This is the raw output of the generated executable _build/default/.test.inline-tests/run.exe but I suppose there is some terminal control command on top of it

random seed: 433741396
Ran: 1 tests in: 0.00 seconds.                                        
SUCCESS

Actual Behavior

$ dune test
qtest .test.inline-tests/run.ml-gen
Extraction : `mod.ml' Done.
         run alias runtest
random seed: 252891362
 [1 / 1] >x>mod.ml:4 *                                                         
 [1 / 1] >x>;32;1mSUCCESS

Reproduction

The files involved:

dune-project:
(lang dune 2.0)
(name "test")

dune:
(library
 (name test)
 (wrapped false) ; The test runner will not compile without that
 (inline_tests
  (backend qtest.lib)))

mod.ml:
let x = 4
(*$T x
    x = 4
*)

Specifications

Version of dune (output of dune --version): 2.5.1
Version of ocaml (output of ocamlc --version): 4.09.1
Operating system (distribution and version): Archlinux on linux 5.6.10
Ounit version: 2.2.2
Qtest version: 2.10.1

Additional information

Link to gist with verbose output (run dune with the --verbose flag): https://gist.github.com/CuiCui66/d0199494627f5120764046a0ed3b63bb

ghost commented 4 years ago

Dune buffers the output of commands so that when running multiple commands in parallel their output is not mixed up.

If you want the command to have direct access to the terminal, you can pass --no-buffer. In this case, you might also want do disable parallel compilation to avoid the output of different commands from being mixed up. So in the end you want to do:

$ dune test -j 1 --no-buffer

This makes me think that we could add a way for an action to declare that it wants to "lock" the terminal. Dune wouldn't buffer such actions, and only one such action could run at a time. While such an action is running, messages coming from Dune would be put in a queue and only printed once the action has completed.

tperami commented 4 years ago

Thank you for the --no-buffer option, it is a nice workaround for now. But having a lock on the terminal would allow command like qtest output to run normally. Another problem with such use case of dune as front for other command is the lack of ".PHONY" aliases. Here the test will run only if some code has changed. It would be nice to have a way to force the test to run again even if no code has changed (in particular for randomized test). Do you think I should open another issue for that?

ghost commented 4 years ago

For .PHONY targets, you can depend on (universe). It means that you depend on everything, so Dune will never cache the rule and will systematically re-execute it.

But having a lock on the terminal would allow command like qtest output to run normally.

Indeed. If you are interested to contribute, I'm happy to give you some pointers :)

tperami commented 4 years ago

For .PHONY targets, you can depend on (universe). It means that you depend on everything, so Dune will never cache the rule and will systematically re-execute it.

I did not know that one, Thank you!

Indeed. If you are interested to contribute, I'm happy to give you some pointers :)

In the idea, why not? But not right now, maybe in a few weeks

ghost commented 4 years ago

Sounds good, I wrote a RFC for this idea: #3464

rgrinberg commented 1 year ago

@tperami is this still a problem? I think we're correctly interpreting the ansi codes emitted by qtest now.

ocaml / dune