jasmine / jasmine-npm

A jasmine runner for node projects.
MIT License
376 stars 145 forks source link

Configure default timeout via flag or environment variable #182

Open dgp1130 opened 3 years ago

dgp1130 commented 3 years ago

Current Behavior

The only ways to set test timeout currently are to explicitly set jasmine.DEFAULT_TIMEOUT_INTERVAL = // ... in JavaScript, or to add a timeout to the specific test callback. When trying to change the timeout for a large number of tests, it requires either modifying every test individually, or having some piece of shared code where the default value can be set. If the jasmine binary is sharded and executed multiple times, there may not be a single place to set this value.

Possible Solution

It should be possible to configure jasmine.DEFAULT_TIMEOUT_INTERVAL via an environment variable or command line flag. Running jasmine --default-timeout 10000 # ... or JASMINE_DEFAULT_TIMEOUT="10000" jasmine # ... should use the provided value as the default timeout.

If any tests override this value directly in code, those should probably take precedence. This flag merely sets the default timeout value for all tests.

Context

This is a particular challenge with Bazel, where tests are often sharded and executed across multiple jasmine executions for higher parallelism and to reduce dependencies on each execution for better cacheability. In this system, a single command like bazel test //... may run many jasmine binaries to run many tests. With this system, it is very difficult to modify the timeout for all the executed tests. Answering a question like "Are my tests running too slowly, or are they stuck altogether?" is much harder without being able to easily increase the test timeout.

Another example I'm encountering right now when debugging flaky tests, I would like to re-run them many times over to reproduce rare bugs. Bazel supports this with --runs_per_test 100, which will run all tests 100 times, helping to reproduce rare bugs. There is also a --jobs flag to run many of these targets in parallel. However in practice, running so many tests in parallel means there is more resource competition and the tests run slower as a result. This causes a number of timeout errors that I can't easily ignore. If I could just set the default timeout via environment variable or command line, I would be able to configure Bazel to pass that through and, then these erroneous failures could be easily ignored for local development via some kind of --define JASMINE_TIMEOUT="99999". Unfortunately, Jasmine provides no such mechanism, and there is no single location where I could add jasmine.DEFAULT_TIMEOUT_INTERVAL = 99999, which would apply to all tests.

Your Environment

slackersoft commented 3 years ago

This sounds like a great feature to have added to Jasmine. I would be happy to review a pull request to add configuration in the node.js library as well as here to allow the current setting to be configured from the external environment.