boxblinkracer / cypress-testrail

A decoupled TestRail integration for your Cypress project to easily send results to TestRail
MIT License
33 stars 24 forks source link

ignorePending not excluding pending test cases from testrail test run results #62

Open chrisgray37 opened 5 months ago

chrisgray37 commented 5 months ago

I am using this in conjunction with cypress/grep. All the pending test cases are because they were skipped due to tagging. However, they are still showing up as untested in testrail run.

I am on this version of cypress-testrail:
"cypress-testrail": "^2.10.0",

Here is my cypress.config.js:

const {defineConfig} = require('cypress');
const TestRailReporter = require('cypress-testrail');

module.exports = defineConfig({
    video: true,
    reporter: 'cypress-multi-reporters',
    reporterOptions: {
        configFile: 'reporter-config.json',
    },
    scrollIntoView: {
        offset: {
            top: -150,
        },
    },
    viewportWidth: 1800,
    viewportHeight: 1000,
    chromeWebSecurity: false,
    e2e: {
        setupNodeEvents(on, config) {
            new TestRailReporter(on, config).register();
            require('@cypress/grep/src/plugin')(config);
            return config
        },
    },
    env: {
        grepFilterSpecs: true,
        grep: {
            showPending: true,
            showSkipped: true,
        }
    },
});

require('@applitools/eyes-cypress')(module);

And here is my testrail section of the cypress.env.json:

  "testrail": {
      "domain": "example.testrail.io",
      "username": "example@example.com",
      "password": "examplePW",
      "projectId": "42",
      "milestoneId": "",
      "suiteId": "",
      "runName": "Cypress Automation Pipeline Run: (__datetime__)",
      "ignorePending": true,
      "closeRun": true,
      "screenshots": true
  },

And here is my console when running the tests:

 1 passing (3s)
  16 pending

 Updating TestRail run R7088. Adding test cases: 9425, 9426, 9427, 9428, 9438, 9429, 9430, 9431, 9432, 9433, 9434, 9435, 9437, 9551, 9552, 9553, 9436 
 TestRun updated in TestRail: 7088 
 Ignoring pending test: C9425: Test Actuator Info 
 Ignoring pending test: C9426: Test Actuator Health 
 Ignoring pending test: C9427: Test Copy Test Data 
 Ignoring pending test: C9428: Test Generate, Reset, Then Delete Test Data 
 Ignoring pending test: C9438: Test Extract 
 Ignoring pending test: C9429: Test Details 
 Ignoring pending test: C9430: Test Resume 
 Ignoring pending test: C9432: Test Restore Collection 
 Ignoring pending test: C9433: Test Move Assets 
 Ignoring pending test: C9434: Test Delete Job 
 Ignoring pending test: C9435: Test List Jobs 
 Ignoring pending test: C9437: Test Find Job 
 Ignoring pending test: C9551: Test Delete Asset - Bad collectionName 
 Ignoring pending test: C9552: Test Delete Asset - Bad profileName 
 Ignoring pending test: C9553: Test Delete DFC 
 Ignoring pending test: C9436: FFV E2E 
 TestRail >> Sending case results to run R7088: C9431 
 Results sent to TestRail R7088 for: C9431 

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        17                                                                               │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      16                                                                               │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     2 seconds                                                                        │
  │ Spec Ran:     api-ffv.cy.js                                                                    │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
chrisgray37 commented 5 months ago

@boxblinkracer

chrisgray37 commented 5 months ago

Also, if I change ignorePending to false, all the pending test results are marked as blocked in testrail; and if I just delete the line altogether, I get the same result as if i did "ignorePending": true,, where it shows all pending tests as untested