semgrep / testo

Test framework for OCaml
ISC License
16 stars 1 forks source link

Keep the test name in a file and use it when reporting dead snapshots #66

Closed mjambon closed 4 months ago

mjambon commented 4 months ago

Tests that capture stdout or stderr output as files ("snapshots") now keep a file named name that contains the test name. When a test suite runs, testo checks for snapshot folders that no longer belong to a known test (which happens when renaming or deleting tests). These dead snapshot folders are now reported as follows:

1 folder no longer belongs to the test suite and can be removed:
  tests/snapshots/testo_tests/1f85f39d5e9a user output capture

The test name user output capture is extracted from the file tests/snapshots/testo_tests/1f85f39d5e9a/name.

Other ways to work with dead snapshots include:

$ cat tests/snapshots/testo_tests/1f85f39d5e9a/name
user output capture

or

$ grep capture tests/snapshots/testo_tests/*/name
tests/snapshots/testo_tests/0048917873df/name:auto-approve > capture stderr
tests/snapshots/testo_tests/02ac0ea4ae90/name:auto-approve > capture stdout and stderr
tests/snapshots/testo_tests/17ec149855c2/name:auto-approve > capture stdxxx
tests/snapshots/testo_tests/1f85f39d5e9a/name:user output capture
tests/snapshots/testo_tests/9c96a5aa8b4b/name:auto-approve > capture stdout

This takes care of #58 and #59. I don't know if we're going to need a special subcommand or option to delete dead snapshots. We can add it later if rm-ing these folders by hand turns out to be tedious.

PR checklist:

Check out CONTRIBUTING.md for more details.