search

anotherchrisberry / spock-retry

Spock extension to allow test features and specifications to be retried before failed
MIT License
34 stars 15 forks source link

Tests are not rerun if failure because of WaitTimeoutException #3

Open levsa opened 9 years ago

levsa commented 9 years ago

Is it correct that timeout failures aren't caught by this extension and retried? I get a test that fails with

geb.waiting.WaitTimeoutException: condition did not pass in 20.0 seconds (failed with exception)

which does not cause reruns.

anotherchrisberry commented 9 years ago

It should be catching and retrying those tests. Which version of Spock are you running? The 0.1 release should work with Spock 1.0, but if you're using 0.7, you'd have to use the code in the initial commit.

levsa commented 9 years ago

Using spock 1.0. ons 26 aug 2015 kl. 17:36 skrev Chris Berry notifications@github.com:

It should be catching and retrying those tests. Which version of Spock are you running? The 0.1 release https://github.com/anotherchrisberry/spock-retry/releases/tag/0.1 should work with Spock 1.0, but if you're using 0.7, you'd have to use the code in the initial commit.

— Reply to this email directly or view it on GitHub https://github.com/anotherchrisberry/spock-retry/issues/3#issuecomment-135067880 .

levsa commented 9 years ago

Just realized the timeout happens in an "at" section called from a base spec setup method. ons 26 aug 2015 kl. 17:39 skrev Levon Saldamli levon.sa@gmail.com:

Using spock 1.0. ons 26 aug 2015 kl. 17:36 skrev Chris Berry notifications@github.com:

It should be catching and retrying those tests. Which version of Spock are you running? The 0.1 release https://github.com/anotherchrisberry/spock-retry/releases/tag/0.1 should work with Spock 1.0, but if you're using 0.7, you'd have to use the code in the initial commit.

— Reply to this email directly or view it on GitHub https://github.com/anotherchrisberry/spock-retry/issues/3#issuecomment-135067880 .

anotherchrisberry commented 9 years ago

Ah, yeah, looks like it doesn't re-run setupSpec methods in superclasses.

Shouldn't be that hard to implement, but I probably will not get to it until this weekend.

levsa commented 9 years ago

I think the retry code even doesn't catch errors in setup methods? Here is the full stack trace:

geb.waiting.WaitTimeoutException: condition did not pass in 20.0 seconds (failed with exception)
    at geb.waiting.Wait.waitFor(Wait.groovy:124)
    at geb.waiting.WaitingSupport.doWaitFor(WaitingSupport.groovy:109)
    at geb.waiting.WaitingSupport.waitFor(WaitingSupport.groovy:67)
    at geb.waiting.WaitingSupport.waitFor(WaitingSupport.groovy:63)
    at geb.Page.waitFor(Page.groovy:81)
    at com.iipax.geb.pages.LoggedInPage._clinit__closure1(LoggedInPage.groovy:7)
    at com.iipax.geb.pages.LoggedInPage._clinit__closure1(LoggedInPage.groovy)
    at geb.Page.verifyThisPageAtOnly(Page.groovy:187)
    at geb.Page.verifyAt(Page.groovy:146)
    at geb.Browser.methodMissing(Browser.groovy:203)
    at geb.navigator.NonEmptyNavigator.click(NonEmptyNavigator.groovy:467)
    at geb.content.TemplateDerivedPageContent.click(TemplateDerivedPageContent.groovy:111)
    at com.iipax.geb.LoggedInGebSpec.setup(LoggedInGebSpec.groovy:25)
Caused by: Assertion failed: 

$("#header-user-menu .ii-logout-link")
|
[]

    at com.iipax.geb.pages.LoggedInPage._clinit__closure1$_closure3(LoggedInPage.groovy:7)
    at com.iipax.geb.pages.LoggedInPage._clinit__closure1$_closure3(LoggedInPage.groovy)
    at geb.waiting.Wait.waitFor(Wait.groovy:113)

LoggedInGebSpec is the base class of my current Spec, and the timeout occurs in its setup()

levsa commented 9 years ago

I tried to fix this but no success. There are interceptors in SpecInfo, e.g. iterationInterceptor, but failures from tests does not propagate to them. There is however setupInterceptors which might work, but then one has to retry each setup method on its own, which wouldn't work.

anotherchrisberry commented 9 years ago

Can you give this a shot and see if it handles the issues you're seeing? https://github.com/anotherchrisberry/spock-retry/pull/5

levsa commented 9 years ago

It does catch the error and retries, but doesn't cleanup and setup everything from start, i.e. no navigation happens while retrying.

I guess the problem is to know which cleanup and setup-methods have been run before the failure and what should be done before rerunning.

I've solved the shakyness that caused the failure in the setup for now, so this can wait for a better solution from spock framework.