Currently there is no simple way to write a tasty ingredient (ingredient transformer, to be precise), which affects order of test execution. Two common applications are:
Randomize order of tests to detect implicit dependencies between them.
Execute the longest running tests first, maximizing multithreaded CPU utilization.
I do not propose to implement such mechanisms in tasty itself, we should keep the core minimalistic. The missing part, which inhibits third parties from providing such functionality, is to provide a way to configure the order of [TestAction] in launchTestTree:
One could extend launchTestTree with an argument (OptionSet -> [TestAction] -> [TestAction]), but that would require us to expose data TestAction, which I deem undesirable:
Instead I suggest adding an argument of type (OptionSet -> (a -> Path) -> [a] -> [a]: "if you tell me how to extract Path from a, I'll be able to reorder [a]".
Currently there is no simple way to write a
tastyingredient (ingredient transformer, to be precise), which affects order of test execution. Two common applications are:I do not propose to implement such mechanisms in
tastyitself, we should keep the core minimalistic. The missing part, which inhibits third parties from providing such functionality, is to provide a way to configure the order of[TestAction]inlaunchTestTree:https://github.com/UnkindPartition/tasty/blob/f73ce423202cab0b4e4d403e7b177d537936b1b8/core/Test/Tasty/Run.hs#L605-L610
One could extend
launchTestTreewith an argument(OptionSet -> [TestAction] -> [TestAction]), but that would require us to exposedata TestAction, which I deem undesirable:https://github.com/UnkindPartition/tasty/blob/f73ce423202cab0b4e4d403e7b177d537936b1b8/core/Test/Tasty/Run.hs#L304-L313
Instead I suggest adding an argument of type
(OptionSet -> (a -> Path) -> [a] -> [a]: "if you tell me how to extractPathfroma, I'll be able to reorder[a]".How does it sound?