[Feature] Show test steps `test.step` in the `list` report (and on assertion fails)

[odinho]

Given the following test file:

import { expect, test } from '@playwright/test';

test('My tests A', async ({ page }) => {
  await test.step('Step 1', async () => {
    expect(true).toBe(true);
  });
});

test('My tests B', async ({ page }) => {
  await test.step('Step 1', async () => {
    expect(true).toBe(true);
  });

  await test.step('Step 2', async () => {
    // several lines
    // of code here
    expect(true).toBe(false);
  });
});

The output of the failed assertion does not mention 'Step 2' in any place:

Running 2 tests using 2 workers

  ✓  1 [chromium] › web/test.spec.ts:3:1 › My tests A (39ms)
  ✘  2 [chromium] › web/test.spec.ts:9:1 › My tests B (42ms)

  1) [chromium] › web/test.spec.ts:9:1 › My tests B ================================================

    Error: expect(received).toBe(expected) // Object.is equality

    Expected: false
    Received: true

      15 |     // several lines
      16 |     // of code here
    > 17 |     expect(true).toBe(false);
         |                  ^
      18 |   });
      19 | });
      20 |

        at step (file:///home/odin/Work/2022-tana/prototype/e2e/web/test.spec.ts:17:18)
        at file:///home/odin/Work/2022-tana/prototype/e2e/web/test.spec.ts:14:14

  1 failed
    [chromium] › web/test.spec.ts:9:1 › My tests B =================================================
  1 passed (474ms)

I am expecting it to say something like (Notice the new "Step 2" at the end):

  1) [chromium] › web/test.spec.ts:9:1 › My tests B › Step 2 =======================================

Instead, to show me the step where it failed. It could also be helpful to show that info in the summary. Especially if using soft assertions and several steps failed in one test. Like:

  ✓  1 [chromium] › web/test.spec.ts:3:1 › My tests A (39ms)
  ✘  2 [chromium] › web/test.spec.ts:9:1 › My tests B (42ms)
       ✘ Step 2
       ✘ Step 3

[jjeff]

Agreed. I'm not quite sure the purpose of test.step() if we don't get a report of what step was being tested when a test fails. Especially for a stress test like this:

test(`Try doing something 20 times`, async () = {
  for (i = 0; i < 20; i++) {
    await test.step(`try #${i + 1}`, async () => {
      // test something
    })
  }
})

It would be really nice to know if the test failed on the first step or some subsequent step.

[amauch-adobe]

Agreed. Where the HTML report uses the test.step to help improve debugging and readability of results it would be nice to have it with the list report since CI's can be configured to not use the HTML report.

[joegoldbeck]

Similarly, we're using the Github reporter, and this just came up as a feature request on our team in order to more quickly isolate failures.

Of course we can always download and open the html report, or open an editor to show the failing line, but that's slower when potentially diagnosing an incident :). Would be great to see the failed step right in the CI output.

[juliachruszczowaH]

Hello-helllo :wave: For me issue is reproducible when running Playwright v1.41.2 with list reporter on GitLab - test.step titles are not listed

(image mcr.microsoft.com/playwright:v1.41.2-focal) in version 1.38.0 step titles were listed on GitLab

[scottwalter-nice]

I don't see the step listed in the error output:

[didu9898]

Also with reporter ['list', { printSteps: true }], even though the steps titles are shown in red color if failed but only first failure details are shown even though multiple steps have status=failed.