joaquimcid / mb-unit

Automatically exported from code.google.com/p/mb-unit
0 stars 0 forks source link

Extending Gallio for Retry #913

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
I've spent the last couple days trying to extend Gallio to allow repeats for 
our GUI automation.  I can get the tests to repeat using the repeat attribute 
however I'm having a tough time swallowing the errors.

For example, If a test is set to retry twice before failing:

Run #1: Fails
Run #2: Passes

Overall result in Icarus is passing, but the overall result in Gallio.Echo is 
failing.  Icarus is producing the desired results.

Icarus is looking at the return outcome from 
IPatternScope.TestBuilder.TestInstanceActions.RunTestInstanceBodyChain and 
showing passing.  However, Gallio.Echo is ignoring this outcome.  To get around 
this issue I used a TestRunner extension that would modify the Statistics from 
the report so that it would show failures as inconclusive unless it's the final 
run of the retry, which is working well for me. 

But the Repeat attribute and the TestRunnerExtension can't talk so it's 
impossible for me to notify the extension when the final run is so I'm forced 
to hard code the number of runs in both areas.  I tried modifying the Test 
MetaData, and parameters, and adding Context MetaData but the 
PatternTestInstanceState is different from the State class inside the runner 
which makes it seemingly impossible to send values.

Is there something I'm missing or would it be possible to add some kind of 
functionality where I can keep some kind of state and make my use-case work?

Final Use-case: I would like to retry tests X number of times until they pass 
or until X is hit.  Any failures before X is hit are ignored if the test 
passes.  If the test does not pass after X times then cause a failure.

Thank you

Original issue reported on code.google.com by mmattera...@gmail.com on 12 Mar 2013 at 8:43