search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47.01k stars 3.18k forks source link

Pending tests are reported as Passed by teamcity reporter #18319

Closed papatelst closed 1 year ago

papatelst commented 3 years ago

Current behavior

We use --env grepTags=@SMOKE along with following parameters with docker run command cypress run --reporter teamcity. Per documentation, teamcity reporter is built into cypress. Teamcity(TC) buildlog shows tests under Pendingand Passingstatus correctly as shown below image However, the TC overviewtab shows all Pending tests as passed as seen below image The TC Tests tab shows all tests as successful. When I filter by 'ignored' status, it does not return any tests as shown below. image Note: TC does not have test filter statuses like Pending, which are mocha specific as mentioned here. The only default filters TC has is 'successful', ''failed' and 'ignored'.

This issue is also reported in Mocha teamcity reporter project here and there seems to be an fix for it here. Differences b/w mocha teamcity reported and cypress using mocha teamcity reporter can be found here.

Desired behavior

Pendingtests from cypress teamcity reporter should display as Ignoredtests in Teamcity UI. It should not display under 'Successful'/Passed status on TC UI.

Test code to reproduce

Dont have

Cypress Version

8.3.0

Other

Relevant plugins included in our project "mocha-teamcity-reporter": "^4.1.0" "cypress-grep": "^2.5.3"

Narretz commented 3 years ago

If it only happens with cypress-grep, you should open an issue in the repository for it: https://github.com/cypress-io/cypress-grep

papatelst commented 3 years ago

@Narretz cypress-grep is only a means to filter and run tagged tests. It is doing its job to run only tagged tests and marking other tests as 'Pending' as seen in Build log snapshot. There is discrepancy in how TC is reporting it on TC UI (passed instead of ignored) I also logged it as Teamcity Issue to see if this is something they can fix.

papatelst commented 2 years ago

Its fixed in mocha-teamcity-reporter using this PR. A similar fix should be applied to cypress project as well.

BlueWinds commented 2 years ago

Thank you for the detailed report, and all the helpful context around it, especially the link to the Teamcity PR where they fixed the issue upstream.

Given that the problem and solution are already known, a pull request would be welcome. I'll add this issue for discussion with the team, but can't promise when we'll get to it.

cypress-app-bot commented 1 year ago

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

cypress-app-bot commented 1 year ago

This issue has been closed due to inactivity.