Allow custom metadata to enable integration with test management system?

[Songyu-Wang]

Current behavior:

No way to add custom metadata to test cases/files that allows users to further process the test results. An example would be tagging test case with ticket number in a test management tool so later user can sync test result under the test ticket.

Desired behavior:

Some sort of tagging system that can be carried to test execution result reporting file(s)

Test code to reproduce

Versions

[jennifer-shehane]

Would something like this feature meet your requirements? https://github.com/cypress-io/cypress/issues/1865

I'd like some better idea on what you expect this would look like or other testing tools that have provided this?

[Songyu-Wang]

1865 is a good starting point, however, it is not sufficient for the following reasons:

1. All the info/data are clustered in test names/titiles which can make them long and less readable.
2. This issue's workaround only solves running selected tests, but we still need to manually parse out the text in the result files to manage which test is mapping to which tag

One example of this feature can be found https://devexpress.github.io/testcafe/documentation/guides/basic-guides/organize-tests.html#specify-test-metadata

TestCafe allows you to specify additional information for tests in the form of key-value metadata. You can display this information in the reports and use it to filter tests.

In my option, display this information in the reports and use it to filter tests natively are the MVP of the feature I am looking for.

In a perfect world, this feature shall have integration with major test management systems, you can find the idea here: https://confluence.xpand-it.com/display/XRAY/Integrating+with+Testing+Frameworks Using one of their examples: https://confluence.xpand-it.com/display/XRAY/Taking+advantage+of+Robot+XML+reports you can find

*** Test Cases ***
Gherkin Valid Login
[Tags]      WEB-1  WEB-3 

They tag the test case with WEB-1 and WEB-2

and in test report:

     <tags>
      <tag>WEB-1</tag>
      <tag>WEB-3</tag>
    </tags>

They are linked to the testcase

and the test report can be directly imported into Xray system and Xray will update test execution to pass/faill according to tag

[Songyu-Wang]

@jennifer-shehane Let me know if there is anything I can clarify

[jennifer-shehane]

@Songyu-Wang Thanks for taking the time to write this out. This definitely clarifies the feature you're asking for. We haven't had anyone ask for this exactly. We'll leave this open as a feature request and evaluate interest from the community based on comments and 👍 here.

[yannbf]

This is an essential feature when using solutions to tag tests. We have a dashboard which provides an overview of our reports and it would be optimal to have a way to add metadata to the tests so that we can filter them out based on our specific configuration. 👍 for this feature

[bdinesh]

Providing an explicit option to add metadata to a test and test suite is a really important feature. Adding this data in the test or test suite title will make them unmanageable. @Songyu-Wang thanks for bringing this up.

For now, will try using the @bahmutov plugin to organize my tests.

+1 for adding this feature

[AndreVirtimo]

+1 Need this as well. I want to integrate my tests with a feature matrix. Currently I have to place tags in the test name, which looks not so good.

[Roemer]

This would really help. For example nunit supports this with Property attributes, so for example:

[Property("TestCaseId", "001")]
[Test]
public void MyTest()
{
...
}

[Roemer]

I created a small helper that allows to add additional data to the it and describe function so that it looks like:

describe("My First Test", { meta: { "data1": "abcd" } }, function() {
  it("Does not do much!", { meta: { "testid": "12345" } }, function() {
    expect(true).to.equal(true);
  });
});

This is fine so far but now I lack possibilities to use that data. I did not find out how to use that data in the test itself (not so important) but I also did not find out how to pass that data to a custom report.

[Roemer]

After a lot more investigation, we found out that cypress actually already supports additional options per test. So something like:

  it("MyTest", { "testid": "12345" }, function() {
    expect(true).to.equal(true);
  });

does work out of the box! Now the only thing you need is a custom reporter which can read all those data like this:

runner
  .once(EVENT_RUN_END, () => {
    this.runner.suite.suites.forEach((suite) => {
        suite.tests.forEach((test) => {        
          const testId = test._testConfig.testConfigList[0].overrides.testid;
        });
    });
  });

[yozlet]

@Roemer Where did you see that Cypress supports this middle argument to the it() function, or are you talking about something else? From what I can tell, Cypress just uses mocha out of the box, which doesn't support that options parameter.

[hkk2018]

@Roemer Where did you see that Cypress supports this middle argument to the it() function, or are you talking about something else? From what I can tell, Cypress just uses mocha out of the box, which doesn't support that options parameter.

https://docs.cypress.io/guides/core-concepts/writing-and-organizing-tests#Test-Configuration