mdashti
commented
3 years ago Similarly: is there a way that I can choose to run only one test out of a test suite?
Open mdashti opened 3 years ago
mdashti
commented
3 years ago Similarly: is there a way that I can choose to run only one test out of a test suite?
mdashti
commented
3 years ago @rcurtin That's already available via test filtering mechanism. Let's say, you see this test failed on Hydra: Delve/Integration/Basic/basic-ml/gdp-life-satisfaction-mse
Then you can run it via XUnit.runtests("test/runtests.jl", "Delve/Integration/Basic/basic-ml/gdp-life-satisfaction-mse").
Generally, you can call it via XUnit.runtests("test/runtests.jl", ["PATTERN1", "PATTERN2", ...])
Patterns are PCRE Regexes: https://www.debuggex.com/cheatsheet/regex/pcre
rcurtin
commented
3 years ago Thanks @mdashti for that tip! :+1:
Before we refactored the testing infrastructure, part of my debugging process was to run the tests until they failed at the first test, inspect the stacktrace, and then fix that issue, slowly moving through the tests. However, this is not the new behavior of the testing framework; now, all tests run, regardless of exceptions being thrown. If any tests fail, stacktraces will be issued for all of them.
That is certainly the desired behavior on Hydra (since you want to see all the tests that failed), but in the Julia REPL this actually is a really bad workflow, since finding the first stacktrace can be pretty hard.
Is there a way that we can set the testing infrastructure up so that when tests are run from inside a REPL, by default we choose to stop testing at the first exception?
In addition, the output from a test often ends up separated from exceptions. Consider this example:
In this example, I have printed the output
from inside the test, in order to do some debugging, but the stacktrace associated with this test (that happens immediately after the
[WARN ]output from mlpack) is much further down. This makes it harder to read the logs and figure out what is going on---is there a way to avoid scheduling all of the other tests until the currently running one is complete?