Pending or Skipped tests not uploaded with the execution

mar1ogz commented 3 weeks ago

Hello :)

Description

I have tested adding tags such as @skip or @only to run only the scenarios I want. However, after the execution is complete, tests with the status pending or skipped are not included in the execution in Jira.

I would like to have the option to include those tests in the execution even if I skipped them manually.

I have tried to use optional settings for the status, but I am using the default settings

In my previous project I had an older version of this plugin and I was not using Cucumber and it was possible to upload the Skipped tests I had disabled using xit(...)

Cypress version

13.10.0

Plugin version

7.0.1

Jira/Xray type

Cloud

Configuration

import { addCucumberPreprocessorPlugin } from '@badeball/cypress-cucumber-preprocessor';
import { configureXrayPlugin, syncFeatureFile } from 'cypress-xray-plugin';
import { defineConfig } from 'cypress';
import coverageWebpack from './cypress/coverage.webpack';
import { DASHBOARD, RESULT } from './cypress/properties/credentials';
import { EnvUrl } from './cypress/properties/env-url.enum';

// eslint-disable-next-line @typescript-eslint/no-var-requires
const { Client } = require('pg');

import createBundler from '@bahmutov/cypress-esbuild-preprocessor';
import createEsbuildPlugin from '@badeball/cypress-cucumber-preprocessor/esbuild';
// eslint-disable-next-line @typescript-eslint/no-var-requires
const fix = require('cypress-on-fix');

// eslint-disable-next-line @typescript-eslint/no-explicit-any
const connections: Record<string, any> = {
  atDashboard: {
    host: DASHBOARD.host,
    user: DASHBOARD.user,
    password: DASHBOARD.password,
    database: DASHBOARD.database,
    port: DASHBOARD.port,
  },
  atResult: {
    host: RESULT.host,
    user: RESULT.user,
    password: RESULT.password,
    database: RESULT.database,
    port: RESULT.port,
  },
  // You can add more data bases repeating the same structure
};

export default defineConfig({
  component: {
    devServer: {
      framework: 'angular',
      bundler: 'webpack',
      webpackConfig: coverageWebpack,
    },
    specPattern: '**/*.cy.ts',
    setupNodeEvents(on, config) {
      // eslint-disable-next-line @typescript-eslint/no-var-requires
      require('@cypress/code-coverage/task')(on, config);
      // eslint-disable-next-line @typescript-eslint/no-var-requires
      const getCompareSnapshotsPlugin = require('cypress-image-diff-js/plugin');
      getCompareSnapshotsPlugin(on, config);
      return config;
    },
    viewportWidth: 1280,
    viewportHeight: 720,
  },

  e2e: {
    experimentalWebKitSupport: true,
    baseUrl: EnvUrl.URL,
    async setupNodeEvents(on, config) {
      const fixedOn = fix(on);

      fixedOn('task', {
        async queryDb({ dbName, query }) {
          const { host, user, password, database, port } = connections[dbName];
          const client = new Client({
            user: user,
            password: password,
            host: host,
            database: database,
            port: port,
          });
          await client.connect();
          const res = await client.query(query);
          await client.end();
          return res.rows;
        },
      });

      fixedOn('file:preprocessor', async (file: Cypress.FileObject) => {
        await syncFeatureFile(file);
        const cucumberPlugin = createBundler({
          plugins: [createEsbuildPlugin(config)],
        });

        return cucumberPlugin(file);
      });

      // eslint-disable-next-line @typescript-eslint/no-var-requires
      const getCompareSnapshotsPlugin = require('cypress-image-diff-js/plugin');
      getCompareSnapshotsPlugin(fixedOn, config);

      await addCucumberPreprocessorPlugin(fixedOn, config);
      await configureXrayPlugin(fixedOn, config, {
        jira: {
          projectKey: 'XXX',
          url: 'url',
          testExecutionIssueKey: 'XXX-3053',
        },
        xray: {
          uploadResults: true,
        },
        cucumber: {
          featureFileExtension: '.feature',
          uploadFeatures: true,
          prefixes: {
            precondition: 'PRECOND_',
            test: 'TEST_',
          },
        },
        plugin: {
          enabled: true,
        },
      });

      return config;
    },
    viewportWidth: 1920,
    viewportHeight: 1080,
    specPattern: ['**/**/*.feature'],
    requestTimeout: 40000,
    responseTimeout: 30000,
  },
});

Acknowledgements

[X] I have made sure not to include any sensitive data in screenshots or obfuscated all occurrences.
[X] I have made sure not to include any sensitive data in descriptions or replaced all values with placeholders.
[X] I have made sure not to include any sensitive data in code snippets or replaced all values with placeholders.

csvtuda commented 3 weeks ago

Hi @mar1ogz,

thanks for the very detailed issue description! I will look into it, theoretically it should be possible to scan all included scenarios for @skip or @only tags. But I need to check how Xray handles such cases first.

csvtuda commented 2 weeks ago

Sorry for the somewhat late reply. I looked into it and the Cucumber plugin seemingly does not at all include skipped tests in the generated JSON report, which my plugin relies on and uploads to Xray. I have started a discussion in the other repository, which you can follow for now for further updates: https://github.com/badeball/cypress-cucumber-preprocessor/discussions/1203

But to me it looks like skipped tests aren't ever coming back to the generated report, which means that we'll need to find a different solution. Cypress's own run results do include skipped scenarios, which could be leveraged instead.

mar1ogz commented 2 weeks ago

Hello!

Nice :) Thanks for getting involved in this topic @csvtuda!

csvtuda commented 1 week ago

Hey @mar1ogz,

I have just released version 7.1.0 (changelog). This version includes a dependency bump of the cypress-cucumber-preprocessor, which fixes the problem with skipped scenarios not showing up in reports.

However, since the preprocessor skips scenarios by marking all the steps they contain as "skipped", I had to add new options:

xray: {
  status: {
    step: {
      failed: // ...
      passed: // ...
      pending: // ...
      skipped: // ...
    }
  }
}

By default, Xray treats Cucumber step results named "skipped" as TO DO (as far as I've seen). So you'll need to explicitly set skipped steps to the status you want to see, e.g.:

xray: {
  status: {
    step: {
      skipped: "MY SKIPPED STATUS"
    }
  }
}

You can check the documentation for another example.

I also looked at @only. It looks to me like the preprocessor ignores this tag completely and executes all scenarios regardless. Unfortunately, I will not be able to investigate this further in the coming weeks. If my observations are correct, you will need to open an issue with @badeball/cypress-cucumber-preprocessor, as my plugin is not responsible for executing tests/scenarios - only for uploading results.

mar1ogz commented 1 week ago

Hey @csvtuda!

Now it is working fine with the @skip (in lowercase) tag! It is set as TO DO because I don't have the SKIPPED status in Xray set up for my project, but I will definitely ask my admin for it :)

Thank you for your hard work and for solving this issue so quickly!

Qytera-Gmbh / cypress-xray-plugin