I've spent the last couple days trying to extend Gallio to allow repeats for
our GUI automation. I can get the tests to repeat using the repeat attribute
however I'm having a tough time swallowing the errors.
For example, If a test is set to retry twice before failing:
Run #1: Fails
Run #2: Passes
Overall result in Icarus is passing, but the overall result in Gallio.Echo is
failing. Icarus is producing the desired results.
Icarus is looking at the return outcome from
IPatternScope.TestBuilder.TestInstanceActions.RunTestInstanceBodyChain and
showing passing. However, Gallio.Echo is ignoring this outcome. To get around
this issue I used a TestRunner extension that would modify the Statistics from
the report so that it would show failures as inconclusive unless it's the final
run of the retry, which is working well for me.
But the Repeat attribute and the TestRunnerExtension can't talk so it's
impossible for me to notify the extension when the final run is so I'm forced
to hard code the number of runs in both areas. I tried modifying the Test
MetaData, and parameters, and adding Context MetaData but the
PatternTestInstanceState is different from the State class inside the runner
which makes it seemingly impossible to send values.
Is there something I'm missing or would it be possible to add some kind of
functionality where I can keep some kind of state and make my use-case work?
Final Use-case: I would like to retry tests X number of times until they pass
or until X is hit. Any failures before X is hit are ignored if the test
passes. If the test does not pass after X times then cause a failure.
Thank you
Original issue reported on code.google.com by mmattera...@gmail.com on 12 Mar 2013 at 8:43
Original issue reported on code.google.com by
mmattera...@gmail.comon 12 Mar 2013 at 8:43