Allow test duration output to be configured

mbg commented 2 years ago

Ever since tasty-0.10, the durations of individual tests (above the 0.01s threshold) and the overall duration of the test suite are displayed by the consoleTestReporter ingredient. This PR adds a new option which controls which test durations are displayed:

--show-durations=all displays individual and overall test durations. This is the existing behaviour and remains the default option.
--show-durations=overall only displays the overall test duration (in the summary at the end).
--show-durations=tests only displays the durations of individual tests (above the 0.01s threshold).
--show-durations=none displays neither.

I see two applications for wanting to control this behaviour:

For golden tests that want to check the test output itself (e.g. when building libraries that extend tasty's functionality). Disabling the reporting of test durations leads to more stable output.
For teaching purposes, students often end up obsessing over test timings when they are displayed, so being able to disable them for test suites given to students may avoid some anxiety for them.

Bodigrim commented 2 years ago

FWIW an ability to control duration output would be useful for tasty-bench.

andreasabel commented 2 years ago

Please rebase onto master and force-push to try and start CI.

mbg commented 2 years ago

@andreasabel I have rebased on master, but you (or another maintainer) needs to approve the workflow.

andreasabel commented 2 years ago

I don't feel like I can make a judgement on whether this feature should be included. But I observe that no tests are included in this PR; and I think that such would be needed to accept the PR.

mbg commented 2 years ago

Sorry about the lack of tests. I had a look at the other options for the consoleTestReporter when I implemented this new option, but couldn't find any tests for those, so there wasn't a natural place for me to add tests for this PR. The contribution guidelines for this repository also don't state that tests are required, so I thought it would be ok without them.

Let me if you have any preference for where/how they should be added and I will see what I can do when I next have some time to work on this.

andreasabel commented 2 years ago

Let me if you have any preference for where/how they should be added and I will see what I can do when I next have some time to work on this.

Not sure either, since I am only a steward maintainer. There are some behavioral tests in coretests/ (shell scripts e.g.), but I don't know how they work. This is something to find out.

I notice that probably also the README needs to be updated (which is the documentation AFAICS), to include the new option.

Bodigrim commented 1 year ago

Closing as superseded by #373. Thanks for your efforts @mbg!

UnkindPartition / tasty

Allow test duration output to be configured #328