Feature request: Support for associating multiple X-Ray test case issue keys to a single cypress test

[jwohlfahrt]

Hi there 👋 I've found this plugin very useful for integrating with X-Ray, and I have no issues using the plugin as currently implemented. However, it would be a nice feature enhancement to support associating multiple X-Ray test case issue keys to a cypress test script. Some QA teams where I work don't have 1/1 mapping between Cypress tests and X-Ray test cases, and in many cases, a single end to end test incorporates steps from multiple test cases. Before switching to X-Ray, our QA team used Testrails, along with a cypress-testrail-simple plugin, which allowed for multiple Testrail test case ids anywhere in the test description (ex it('C101 C202 works, ...)`). It would be nice if we could bake this functionality into cypress-xray-plugin. Let me know if you have any questions or would like more info. Thanks, and appreciate the work on this plugin so far 😄

[csvtuda]

Hey @jwohlfahrt,

thanks for the suggestion! On paper, I think this should be easy enough to implement. But I have a question:

How should the following situation be handled?

it("C101 C202 works", () => { /* ... */ });

it("C202 fails", () => { /* ... */ });

What should the final test status of C202 be in this situation? Xray only allows setting a single status for each test during results upload (excluding data-driven/iterated tests). Does TestRail (or the mentioned plugin) handle such use cases differently?

[jwohlfahrt]

@csvtuda Thanks for the quick reply! Ideally, I think the status of the corresponding x-ray test case would be set to failed if any cypress test mapped to the test case id fails. I'm not sure how easy it would be to keep track of all of this during the run though. Given the test case single status limitation of X-Ray API, maybe this would just be an unsupported use case for the plugin, if this would be to difficult to track over the entire execution. In our case, we won't have more than one cypress test tied to a single x-ray test case id, so this wouldn't be a scenario we run into at our place of work.

Just out of curiosity, I did test out passing the same test case id to multiple cypress tests with the current plugin implementation, and it seems like the status of the test that executes first takes precedence. So if two tests are marked with the same case id, and the first one passed, but the second test fails, the test case is marked as passing in X-Ray.

[csvtuda]

@jwohlfahrt Okay thanks for the clarification. I think I will consider this change for a later release, seeing that you're currently not waiting for it. It should then probably be accompanied by some sort of conflict resolution configuration:

xray: {
  status: {
    /*
     * What to do when multiple tests linked to the same Jira issue have different test statuses.
     * 
     * @default 'prefer-failure' 
     */
    conflictResolution: 'prefer-failure' | 'prefer-success' | (...statuses: string[]) => string
  }
}

In our case, we won't have more than one cypress test tied to a single x-ray test case id, so this wouldn't be a scenario we run into at our place of work.

Just let me know when this changes, so I can focus on this issue instead of doing the stuff I'm currently working on :+1:

[jwohlfahrt]

@csvtuda Thanks for your reply! Just to clarify, my team does have an immediate need for associating multiple X-Ray test case ids to a single cypress test case (it block), like:

it("EMB-123 EMB-124 EMB-125 example test", () => { /* ... */ });

I would expect the three test cases to get marked with the same status in X-Ray, depending on if the case passes or fails.

We just don't have any cases currently where we'll need to tag multiple cypress tests (it blocks) with the same test case id, like:

it("EMB-123 example test 1", () => { /* ... */ });

it("EMB-123 example test 2", () => { /* ... */ });

No worries if this still needs to be a backlog item, but my team will be ready to consume this one cypress to many x-ray test cases scenario as soon as its available. Hope that helps clarify things a bit. Thanks!

[csvtuda]

Hey @jwohlfahrt,

sorry for the late reply. I have just released version 7.2.0 (changelog). Cypress tests can now contain more than one test issue key:

it("CYP-123 CYP-124 CYP-125 example test", () => { 
  expect(true).to.be.true;
});

Please let me know if everything works as desired.

[jwohlfahrt]

@csvtuda Just saw this, and actively working through trying to test this. Ended up running into a few other issue with 7.x.x that I'm trying to resolve (opened one issue for clarification), but will update once I can test this properly. Thanks!

[jwohlfahrt]

@csvtuda I was finally able to test this out, and its working great. Many thanks for getting this feature incorporated! 🙏

[csvtuda]

Thanks for the feedback! Much appreciated.