search

kuceb / cypress-plugin-retries

A Cypress plugin to retry failed tests.
MIT License
237 stars 20 forks source link

Jenkins test report shows test passed but in actual it failed #19

Closed nikhilesh009 closed 5 years ago

nikhilesh009 commented 5 years ago

when I ran the test on the test runner if a test failed after retires, it shows test failed, but i ran on Jenkins test failed but in the test result, it shows test passed but screenshot generated that shows test failed. is it possible that generated screenshot shows the first attempt failed the test ?

kuceb commented 5 years ago

@nikhilesh009 is it the last test in the suite (last it in a describe block for example)? There may be some edge-cases where the logic is off on knowing whether a test will re-run. I have changes in cypress-core that fixes this, but not sure it can be fixed in this plugin

nikhilesh009 commented 5 years ago

Yes, its last test in a describe block. is it possible that the test is stuck too at the end?

kuceb commented 5 years ago

@nikhilesh009 I wasn't able to reproduce this after trying different combinations of nested tests with various hooks. I would be happy to look more into this if you have a minimal reproducible spec file that demonstrates the problem.

panli666 commented 5 years ago

hi, i meet same issue in CI, the case was failed, but in report it is pass!

lavender21 commented 5 years ago

Also, have the same issue in CI, we use buildkite. Any update about this issue?

nikhilesh009 commented 5 years ago

I was concern about test passing/failing, so i intentionally failed my test and then test on Jenkins, On Jenkins test report it shows correctly, the test failed, and in output 3 screenshot of the failed test. but still, i am not sure its always the case.I am testing always third parties supplier, and sometimes it's up-down.

when I ran the test on the test runner if a test failed after retires, it shows test failed, but i ran on Jenkins test failed but in the test result, it shows test passed but screenshot generated that shows test failed. is it possible that generated screenshot shows the first attempt failed the test ?

kuceb commented 5 years ago

Can anyone post the simple spec code that can reproduce this? I don't have time to guess and check dozens of combinations of passes/failures with/out hooks and such.

panli666 commented 5 years ago

Hi @Bkucera I found the issue, when i add after hook to clear the cookies, the case will be marked as pass, even it is failed.

`context('Window', () => { beforeEach(() => { cy.visit('https://example.cypress.io/commands/window') })

it('cy.window() - get the global window object', () => { Cypress.currentTest.retries(2) // https://on.cypress.io/window cy.window().should('have.property', 'top') cy.get('#navbar > :nth-child(1) > :nth-child(2) > a1').click(); }) after(() => { cy.clearCookies(); }); })` image

kuceb commented 5 years ago

@PansyLi awesome, I'll try to reproduce this again

kuceb commented 5 years ago

:tada: This issue has been resolved in version 1.3.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: