badeball / cypress-cucumber-preprocessor

Run cucumber/gherkin-syntaxed specs with Cypress
MIT License
1.32k stars 149 forks source link

How can I retry the scenario or step definition if the test was failed #545

Closed ngocthuan1208 closed 2 years ago

ngocthuan1208 commented 3 years ago

There is the test retries configuration on cypress.json. But in some special situation, I don't want to retry all the tests, only some tests need to be retried.

Can we add the retry on a specific scenario?

phi0411141 commented 3 years ago

Upvoted, I also need a way to add configurations per test scenario

sKavanaghV commented 3 years ago

Cypress does provide native retry: https://docs.cypress.io/guides/guides/test-retries#Custom-Configurations

And this lovely plugin does accept a test config block... Given( "the dashboard is sorted by column {string} in {string} order", { retries: { runMode: 3, openMode: 3, }, }, (column, order) => { ... ...

But it's not working. Tracing the generated test (as seen in the test runner): (0, _steps.Given)("the dashboard is sorted by column {string} in {string} order", { retries: { runMode: 3, openMode: 3 } }, function (column, order) { ... ...

Looks OK but no dice. If I do the same in a spec directly (no cucumber) the retries override does work as expected.

Have not been able to figure out why it's not working under this plugin.

phi0411141 commented 3 years ago

As I dive into the source code, the generated code looks like this

  `describe(${filename}, ()=> {...})`

which will fall to the no-config case of cypress. Actually I can help a pull request for this, but not sure where to put the configs in the feature file

michahell commented 3 years ago

😢 Is there a way to do this from within Cypress steps, as in configure this ad-hoc outside of the actual definition?

postoronnii commented 3 years ago

Retries doesn't work correctly. If I use for ex. retries:2 in cypress.json, and two test scenarios, when the first scenario fails it retries only the second scenario 4 times.

lucassouzaspc commented 3 years ago

@sKavanaghV you get the solution to this problem?

surbina commented 3 years ago

Have you tried: https://docs.cypress.io/guides/guides/test-retries#Configure-retry-attempts-for-all-modes ?

badeball commented 2 years ago

Due to personal reasons, the previous maintainers of this package are stepping down and handing the reigns over to me, a long-time contributor to the project and a user of it myself. This is a responsibility I'm very excited about. Furthermore, I'd like to thank @lgandecki ++ for all the work that they've done so far.

Read more about the transfer of ownership here.

The repository has however moved and all outstanding issues are being closed. This is not a reflection of the perceived importance of your reported issue. However, if after upgrading to the new version, you still find there to be an issue, feel free to open up another ticket or comment below. Please make sure to read CONTRIBUTING.md before doing so.

trprincebautista commented 2 years ago

Any updates on this?

badeball commented 2 years ago

Cypress offer no mechanism to retry a subset of a test, which means retrying a step definition isn't possible. The preprocessor is after all constrained / limited to do what Cypress can offer.

As for retrying specific scenarios, I'm open to suggestions as to how the interface could look like. How would one specify a single scenario to be retried N times?

hect1c commented 2 years ago

Retries doesn't work correctly. If I use for ex. retries:2 in cypress.json, and two test scenarios, when the first scenario fails it retries only the second scenario 4 times.

Before the update this was working for me. I was able to set the retries in the cypress.json file and each scenario would retry accordingly if any step failed. This is currently not working and I am getting the same behavior mentioned above. The step will actually just continue if the first one fails and I believe the last one is being retried N times.


As for retrying specific scenarios can we utilize the Smart Tagging feature and have a @retries:N or better yet an options tag that takes a key=value list something like @options(retries=3, focus=true, tag=mobile) giving a clean way to provide a set of key=value smart tagging options as well as the tag key for filtering tests.

amitguptagwl commented 2 years ago

@ngocthuan1208 @phi0411141 You can probably try cytorus to retry scenario.

badeball commented 2 years ago

Before the update this was working for me. I was able to set the retries in the cypress.json file and each scenario would retry accordingly if any step failed. This is currently not working and I am getting the same behavior mentioned above. The step will actually just continue if the first one fails and I believe the last one is being retried N times.

I recommend opening a separate issue about this and follow the issue template. I can tell you immediately that I need to have a reproducible example that I can clone.

badeball commented 2 years ago

I've create an issue specifically related to the proposed update here and am closing this.