Nodes do not finish a test stage if the last result for a stage is a timeout.
Test packages must make sure that their requests will timeout, otherwise you can run out of sockets before the test run is even finished.
If the test request does timeout, there isn't presently a way for the test package to signal to orca-node that a timeout occurred.
I'd like to have timeouts handled in only one place, rather than in both the orca-node test running code and the test package function, if possible. Having two layers of timeout detection opens us up to race conditions; we'd need to make sure that the timeout isn't reported twice.
One way to approach this is to delegate all timeout detection to the test packages, having them signal a timeout by constructing an agreed-upon error and pass that to the callback. A problem with this approach is that it does not provide a way for orca-node to protect itself against sloppy test package code.
Another approach is to keep the timeout handling in orca-node, but pass the timeout value to the test package, so that it can set appropriate timeouts, BUT the test package will ignore timed out requests rather than trying to report them to orca-node.
I'd like to have timeouts handled in only one place, rather than in both the orca-node test running code and the test package function, if possible. Having two layers of timeout detection opens us up to race conditions; we'd need to make sure that the timeout isn't reported twice.
One way to approach this is to delegate all timeout detection to the test packages, having them signal a timeout by constructing an agreed-upon error and pass that to the callback. A problem with this approach is that it does not provide a way for orca-node to protect itself against sloppy test package code.
Another approach is to keep the timeout handling in orca-node, but pass the timeout value to the test package, so that it can set appropriate timeouts, BUT the test package will ignore timed out requests rather than trying to report them to orca-node.
I'm going to try making the latter work.