Test results of grouped test steps

jerhardt-adesso commented 2 months ago

Description

Hi,

When grouping multiple it functions under a describe block that contains the test issue key some undesired behavior occurs.

Example:

describe('TEST-123', () => { it('should do A', ...) it('should do B', ...) })

My expectation is that if one of those steps fails, then the test result that is going to be uploaded to xray ticket TEST-123 is going to be "failed". Somehow the plugin still would upload the result as "success" if the last step was successful although the first one failed. Currently I have implemented a workaround where I keep track of all failed tests in a test run and check the number to not be greater than 0 otherwise throw an error to create a failure towards the end so that the result is being uploaded correctly as "failed".

Another problem I discovered with my current approach is that screenshots of failed tests are also not being uploaded, because apparently the name does not include the test issue key. Even though the warning message itself clearly shows the test issue key within the file name, it still claims to not be existent. I have even tried to rename the screenshot after it has been taken to follow the warning's proposed naming convention where the key is in the front e.g. "TEST-123 - ...", but unfortunately the problems persists.

Cypress version

13.6.2

Plugin version

6.0.0

Jira/Xray type

Server

Configuration

No response

Acknowledgements

[X] I have made sure not to include any sensitive data in screenshots or obfuscated all occurrences.
[X] I have made sure not to include any sensitive data in descriptions or replaced all values with placeholders.
[X] I have made sure not to include any sensitive data in code snippets or replaced all values with placeholders.

csvtuda commented 2 months ago

Hi @jerhardt-adesso,

thanks for creating the issue and for bringing this up. Both issues are due to the fact that the plugin expects issue keys to be present in the innermost it() and not in one of the surrounding describe() functions.

Another problem I discovered with my current approach is that screenshots of failed tests are also not being uploaded, because apparently the name does not include the test issue key.

At the end of a test run, the plugin receives an array of screenshots taken by Cypress and needs to map them to the executed tests. The only way it can reliably do this is by matching issue keys in a test's title with issue keys found in the screenshots' filenames. Since there is no issue key in your inner it(), the plugin fails to find a matching test for the screenshot and reports the warning. Such grouped tests aren't (yet?) supported by the plugin.

My expectation is that if one of those steps fails, then the test result that is going to be uploaded to xray ticket TEST-123 is going to be "failed". Somehow the plugin still would upload the result as "success" if the last step was successful although the first one failed.

The plugin actually uploads both test results. Here's the request body it sends to Xray when I recreate your example:

"body": {
  "testExecutionKey": "CYP-663",
  "info": {
    "project": "CYP",
    "startDate": "2024-05-07T18:49:33Z",
    "finishDate": "2024-05-07T18:49:38Z",
    "summary": "Execution 07/05/2024, 20:49:29",
    "testEnvironments": ["DEV"]
  },
  "tests": [
    {
      "testKey": "CYP-665",
      "start": "2024-05-07T18:49:33Z",
      "finish": "2024-05-07T18:49:33Z",
      "status": "PASSED"
    },
    {
      "testKey": "CYP-665",
      "start": "2024-05-07T18:49:33Z",
      "finish": "2024-05-07T18:49:34Z",
      "status": "FAILED"
    }
  ]
}

Xray's result import endpoint simply seems to be implemented such that it uses the first encountered status and discards the rest (you should be able to see the status flipping when you switch the order of your tests).

I actually don't yet know how I could possibly implement your use case. How are you splitting your tests compared to what's defined in Xray?

Say, for example you've got the following test steps in Xray test CYP-123:

Click monkey
Click tiger
Feed elephant

Do you then put each step into its own test like this?

describe('CYP-123', () => {
  it('Click monkey', () => { /* ... */ });
  it('Click tiger', () => { /* ... */ });
  it('Feed elephant', () => { /* ... */ });
});

Or are you basically writing data-driven tests?

describe('CYP-123', () => {
  it('displays animals for Jane', () => { 
    loginAsJane();
    clickMonkey();
    clickTiger();
    feedElephant();
  });

  it('displays animals for George', () => { 
    loginAsGeorge();
    clickMonkey();
    clickTiger();
    feedElephant();
  });

The latter could (partly) be modelled using Xray's iterations feature: grafik

As to the former: I don't think this can be modelled without introducing some sort of very complex step matching behaviour.

jerhardt-adesso commented 2 months ago

Your first example fits our current approach pretty well:

describe('CYP-123', () => {
  it('Click monkey', () => { /* ... */ });
  it('Click tiger', () => { /* ... */ });
  it('Feed elephant', () => { /* ... */ });
});

We have been using this approach so far, because we have a lot of different sites and environments where we need to perform the same tests on a regular basis. Grouping all tests that belong to a certain context under a single test in xray seemed more reasonable for us, as it would mean that we don't have to split up every single step into a single test in xray which would increase by the number of sites and environments that are added and have to be mapped to. We are still experimenting with different ideas, but try to keep everything more modular.

Another idea of us would be to have a test issue in xray for a given context with multiple test execution tickets that have different environment values set where the results of the cypress tests for each environment are being uploaded to. Unfortunately, I was not able to set the testExecutionIssueKey dynamically while running the tests, so that the results could be uploaded to the same test execution ticket each time, unless I define the variable before I run the tests.

Edit: We have changed the structure of our Xray Repository so that it can work with the plugin.

csvtuda commented 2 months ago

Edit: We have changed the structure of our Xray Repository so that it can work with the plugin.

I'm very happy to hear that, because I had been thinking about it and wasn't really able to come up with a clean solution.

The only thing I can now still suggest (and which you're probably already employing) is making use of test environments:

Qytera-Gmbh / cypress-xray-plugin