inluxc / playwright-xray

XRAY reporter for Playwright
MIT License
27 stars 11 forks source link

Support data-driven test results upload #90

Open csvtuda opened 14 hours ago

csvtuda commented 14 hours ago

Hello.

Is your feature request related to a problem? Please describe.

I've recently started using this reporter and it's working great for "normal" test cases. However, in my project we're often dealing with data-driven test cases such like this:

for (const name of ["Bob", "George", "Linda"]) {
  test(`ABC-123 | greet ${name}`, async ({ page }) => {
    await page.goto('https://example.org/');
    const title = page.getByRole('heading');
    await expect(title).toHaveText(`Hello ${name}!`);
  });
}

A corresponding, parameterized Xray test case would look like this:

test definition

Its dataset like this:

dataset

And corresponding test executions like this:

test execution


The reporter currently doesn't support such parameterized/data-driven tests and uploads and instead creates a "normal" result:

reuslt

Are there plans to support data-driven tests in the future?

Describe the solution you'd like The solution would probably be to fill out the iterations information in the uploaded Xray JSON.

test: {
  status: 'PASSED',
  testKey: 'ABC-123',
  iterations: [
    { status: 'PASSED', parameters: [ {name: 'name', value: 'Bob'}] },
    { status: 'PASSED', parameters: [ {name: 'name', value: 'George'}] },
    { status: 'PASSED', parameters: [ {name: 'name', value: 'Linda'}] }
  ]
}

The challenge will be finding a way to enhance each test with its parameter information. For this, I can think of two approaches:

  1. Let users encode the iteration parameters in the title and decode them in your reporter:

    test(`ABC-123 name=${name} | greet ${name}`, async () => {
      // ...
    });
  2. Let users attach iteration parameter information and retrieve them in your reporter:

    test(`ABC-123 | greet ${name}`, async ({}, testinfo) => {
      await testinfo.attach('xray-parameters', { body: JSON.stringify({ name: name }, null, 2) });
    });

Describe alternatives you've considered Maybe there's also a way to provide a fixture that does this automatically? I'm not sure though.

Desktop (please complete the following information):