Guidance on providing a way to retry certain tests?

[LaurentRDC]

Hello,

At work we have some tests which are naturally flaky -- maybe 5% of the time, they may fail for known reasons which we cannot account for. We would like to specify a number of retries for these specific tests.

Would anyone here be able to guide me on the best way to hook into Tasty to do this? The goal is to provide a function of the following form:

retryOnFailure :: RetrySettings -> TestTree -> TestTree

and perhaps package it as tasty-retry or something that people can use.

My concrete question is: should this be a Tasty Ingredient or a Tasty Provider? I'm a bit unclear on the ways to extend Tasty

[Bodigrim]

This could be an Ingredient I think. A test provider should be enough:

newtype Retry a = Retry a

instance IsTest a => IsTest (Retry a) where 
  testOptions = [{- define an option to read --retry=N from command line -}]
  run opts (Retry a) yieldProgress = {-  run a, retry if necessary -}

I can elaborate further if needed.

[LaurentRDC]

Thanks for the suggestion, I have created small package here, and have published a package candidate on Hackage

I would love people's feedback before I make a real release

[Bodigrim]

• flakyTest :: (RetryPolicyM IO) -> TestTree -> TestTree seems incomplete, one should account for PlusTestOptions, AskOptions and After.

• It would be nice to use callback to report progress of retrying (but keep status update short).

• It's desirable to keep resultDescription as short as possible. If it overflows one display line, it can mangle terminal output if --hide-successes is enabled. I'd certainly use "μs" instead of "microseconds" and probably shorten up other parts as well.

[LaurentRDC]

Good catch on the completeness of flakyTest! Thanks @Bodigrim .

It was also a good call to make use of the callback -- the test suite of tasty-flaky shows an example of this now.

I think this is good enough for a first release -- thank you for your help