[Question ] Custom Reporter Issue

[jithintill]

Context:

• Playwright Version: 1.22
• Operating System: [e.g. Windows, Linux or Mac]
• Node.js version: [e.g. 12.22, 14.6]
• Browser: [e.g. All, Chromium, Firefox, WebKit]
• Extra: [any specific details about your environment]

I am trying to perform updates onto testrail using custom reporter.

But it seems like updates are not happening async onTestEnd block

Test code and logs attached.

Before finishing off test 1, test 2 and test 3 starts and after that test 1 starts.

If i run one test, sometimes updates doesnot happen to testrail

Code Snippet

import { testRailUpdate } from "../testrail/utils"

export default class CustomReporter implements Reporter {

  onTestBegin(test) {
    console.log(`Starting test ${test.title}`);
  }

  async onTestEnd(test: TestCase, result: TestResult): Promise<void> {
    const testCaseID = test.title
    await testRailUpdate(testCaseID, result.status)
    console.log(`Finished test ${test.title}: ${result.status}`);
  }
}

Describe the bug

Actual Result

Starting test C2657_user_registration_test Starting test C2639_Login/Logout Functionality Starting test C2653_User Profile Verification Finished test C2657_user_registration_test: passed Finished test C2639_Login/Logout Functionality: passed Starting test C2943_Edit_user_test Finished test C2653_User Profile Verification: timedOut

Expected Result Starting test C2657_user_registration_test Finished test C2657_user_registration_test: passed Starting test C2639_Login/Logout Functionality Finished test C2639_Login/Logout Functionality: passed Starting test C2653_User Profile Verification Finished test C2653_User Profile Verification: timedOut Starting test C2943_Edit_user_test Finished test C2943_Edit_user_test: passed

Add any other details about the problem here.

[mxschmitt]

What do you mean with that updates are not happening?

Do you mean that C2943_Edit_user_test does not finish?

Can you provide a small repro for us?

[jithintill]

I have updated expected and actual result. Updates are happening ,but the onTestEnd is not awaiting until the action is complete Before finishing off testrail updated on test 1, test 2 starts and then test 3, after that test 1 updates gets completed.

[rwoll]

onTestEnd (and onTestBegin) is not documented as async (it returns void, not Promise<void>). Playwright Test expects onTestEnd to be sync and will not await it[^1]: https://github.com/microsoft/playwright/blob/d7b63fa0b45ab12e40d0018b0e6f796ecd2df4b7/packages/playwright-test/types/testReporter.d.ts#L426

If you want some code in your onTestEnd to wait for some code in your onTestBegin to finish for a corresponding test you can do queuing or barriers like so:

import type { Reporter, TestCase } from '@playwright/test/reporter';

class TaskQueue {
  private _chain: Promise<any>;

  constructor() {
    this._chain = Promise.resolve();
  }

  postTask(task: () => any): Promise<any> {
    const result = this._chain.then(task);
    this._chain = result.catch(() => {});
    return result;
  }
}

class CustomReporter implements Reporter {
  private _queue: TaskQueue = new TaskQueue();

  onTestBegin(test: TestCase) {
    console.log('onTestBegin FIRED');
    this._queue.postTask(async () => {
      console.log('onTestBegin:hook:START', test.title);
      // simulate this taking a really long time
      await new Promise(r => setTimeout(r, 30_000));
      console.log('onTestBegin:hook:END', test.title);
    })
  }

  onTestEnd(test: TestCase) {
    console.log('onTestEnd FIRED');
    this._queue.postTask(async () => {
      console.log('onTestEnd:hook:START', test.title);
      // simulate this running quickly
      await new Promise(r => setTimeout(r, 500));
      console.log('onTestEnd:hook:END', test.title);
    })
  }

  async onEnd() {
    // explicitly wait for this to finish
    await this._queue.postTask(async () => {});
  }
}

export default CustomReporter;

The output looks like:

$ npx playwright test --grep 'should allow me to display active items'
onTestBegin FIRED
onTestBegin:hook:START should allow me to display active items
onTestBegin FIRED
onTestBegin FIRED
onTestEnd FIRED
onTestEnd FIRED
onTestEnd FIRED
onTestBegin:hook:END should allow me to display active items
onTestBegin:hook:START should allow me to display active items
onTestBegin:hook:END should allow me to display active items
onTestBegin:hook:START should allow me to display active items
onTestBegin:hook:END should allow me to display active items
onTestEnd:hook:START should allow me to display active items
onTestEnd:hook:END should allow me to display active items
onTestEnd:hook:START should allow me to display active items
onTestEnd:hook:END should allow me to display active items
onTestEnd:hook:START should allow me to display active items
onTestEnd:hook:END should allow me to display active items

You can get fancier/more efficient, but this should illustrate the basic concept.

Does this work for you?

[^1]: onEnd (NOT onTestEnd), however, is allowed to be async and PW will await it.

[rwoll]

Closing per-above as part of triage; please re-open if the suggested code does not work for you. Thanks!

[jithintill]

Thanks for this, this is working fine. I think its good to add this to the docs because its helpful for reporting purpose.