Extension with new subcommands: status, approve, rerun

[mjambon]

Hi. I'm looking into adding functionality to our test suite. We'll be porting tests from Pytest to Ocaml and we'll need a convenient way to check stdout and stderr outputs. I'm not sure what the implementation would look like. At best, it extends the command-line interface generated by alcotest without breaking existing code. At worst, it's some custom code in our (open-source) project that nobody else can use conveniently.

Here's what I have in mind:

Subcommands offered by the generated test program

• run (today known as test): run the tests
• list: list the tests - shows the names, identifiers, or tags that can be used to filter tests
• status*: show the status of the last run for each test - OK, FAIL, XFAIL, XPASS, or Unknown
• approve*: approve the output of the last run for the selected tests, similar to dune promote and pytest --snapshot-update without rerunning the tests
• rerun*: rerun the failed tests (and those that haven't run)

* new subcommand

All these subcommands support filters that restrict the selection of tests. Filtering can be done:

• by specifying a regexp (like today) to match the test or test case name
• by specifying an exact substring to match the test or test case name
• by selecting a combination of tags e.g. some_product and not todo

Library changes

• A test definition comes with more options than the tuple (name, quick/slow, test function). New options include:
  • check against expected output on stdout, stderr, or both (separately or interleaved)
  • check against expected output files
  • a list of tags (similar to pytest markers) for selecting/deselecting groups of tests - generalizes the existing quick/slow tagging
  • expected test status: success or failure ("xfail") - essential for writing tests ahead of time
• A compact, unique, stable identifier exists for each test. These IDs can be generated. They're important for selecting individual tests simply and unambiguously.
• Support for the legacy way of creating a test.

Please let me know if there's already something we can use that supports capturing stdout/stderr and the status and approve subcommand.

Other requirements

• Independence from the build system (dune) or from any framework whose primary job is not to manage tests.
• Only one kind of tool/framework/library may be used to manage all test suites for OCaml code.

[mjambon]

I did a bit of work toward these goals. This is a lot of code (~2000 lines), all of it as a layer on top of the Alcotest library. The code, for now, is kept within the semgrep project at https://github.com/semgrep/semgrep/tree/develop/libs/alcotest_ext. Given its size and complexity, it should probably not be merged into the main Alcotest project but I'm still up for it. The current status is "alpha" in the sense that many features will need to be adjusted after we use it. The visual output is a bit of a mess.

Here's what was implemented and works:

• 3 subcommands: run, status, approve. All 3 write or read the status of tests stored on disk.
• "xfail" results = tests that are expected to fail. A completed test is in 1 of 4 classes: PASS, XFAIL, FAIL, XPASS (same terminology as pytest).
• capturing and comparing output against expectations (stdout, stderr, merged stdout/sderr, separate stdout/stderr, none).
• a test is now a record created with a function that takes a bunch of optional arguments. The required arguments are the test name and the test function.
• hierarchical categories. These are nice to keep things organized and easy to filter, as well as to avoid name clashes.
• run --lazy: only reruns the tests that previously failed. Useful when working on fixing a small number of tests.
• unique test IDs: these are hashes of the full test name (category + name) with a low chance of accidental collision. Makes file names meaningless but compact. Allows easy filtering by specifying the test ID e.g. run -s d84498857f5e.

Unchanged:

• the tests are written with Alcotest.check and accompanying functions (or whatever exception-raising checks the user likes).
• running the tests is still delegated to Alcotest. This will probably change so as to make the output less confusing, notably with respect to tests that are expected to fail.

Partially implemented:

• tags equivalent to pytest markers (I just prefer the term "tag"). Filtering by tag isn't implemented yet.
• test filtering: currently, only filtering by substring occurring in the test category, name or ID is supported. --lazy is another form of filtering that's usable. Filtering by tags and combining all this in a unified boolean query language may be a good idea in addition to shortcut options such as --lazy or --tags.
• readability and rich output formatting: needs improvement.
• JUnit output (somewhere in a private project of ours, using https://github.com/Khady/ocaml-junit)

Unavailable:

• many Alcotest options I haven't used, which may or may not be important to some users.
• Lwt support: only the conversion to a regular Alcotest test suite is provided. I have questions about the need for such support.
• Async support

[mjambon]

The project is now known as "Testo" and has its own repo: https://github.com/semgrep/testo. We're using it in Semgrep.