Tests are hanging in an in progress state when retries is greater than 0

[costa-collibra]

Here is the config:

const testTags = process.env.TAGS ? process.env.TAGS : 'All';

const RPconfig = {
  token: process.env.REPORT_PORTAL_API_KEY,
  endpoint: process.env.REPORT_PORTAL_URL,
  project: process.env.PROJECT_NAME,
  launch: `${process.env.BRANCH_NAME} - ${process.env.BUILD_NUMBER}`,
  attributes: [
    {
      key: 'branchName',
      value: process.env.BRANCH_NAME,
    },
    {
      key: 'testTags',
      value: testTags,
    },
  ],
  description: 'E2E Playwright test launch',
};

const reportList: ReporterDescription[] = [
  ['list'],
  ['html', { open: 'never' }],
  ['line'],
];

if (process.env.CI === 'true') {
  reportList.push(['./utils/CiReporter.ts']);
  reportList.push(['@reportportal/agent-js-playwright', RPconfig]);
}

const config: PlaywrightTestConfig = {
  testDir: './tests',
  timeout: 2 * 60 * 1000,
  expect: {
    timeout: 5000,
  },
  fullyParallel: true,
  retries: process.env.CI === 'true' ? 1 : 0, // report portal issue when retries is > 0
  workers: process.env.CI === 'true' ? 5 : 1,
  forbidOnly: process.env.CI === 'true',
  reporter: reportList,
  outputDir: 'test-assets/',
  use: {
    browserName: 'chromium',
    headless: process.env.CI === 'true',
    screenshot: {
      mode: 'only-on-failure',
      fullPage: true,
    },
    video: 'retain-on-failure',
    actionTimeout: 5000,
    navigationTimeout: 30000,
    baseURL: process.env.BASE_URL,
    trace: 'retain-on-failure',
    launchOptions: {
      slowMo: 0,
    },
  },
};

export default config;

Report portal results just hang in progress and have to be force stopped.

However when retries is 0 it works.

Please can someone investigate?

[AmsterGet]

Hello @costa-collibra ! Could you please specify the versions of Playwright and agent-js-playwright you are using? I'll try to take a look.

[costa-collibra]

Hi @AmsterGet

Yes of course I am using:

"@playwright/test": "1.31.0", "@reportportal/agent-js-playwright": "5.0.8",

Thanks for investigating

[costa-collibra]

The 3 dots show the in progress state which is stuck permanently

[AmsterGet]

@costa-collibra Did you have any errors in the console?

[costa-collibra]

I dont see any errors

in this screenshot you can see 2 runs 1 with retries and 1 without. 1 completes and the other is stuck

Here is the console output

Running 6 tests using 1 worker

  -  1 browseComponent.test.ts:13:8 › Browse panel › @p0e2e User opens Browse panel and searches for community
  ✓  2 browseComponent.test.ts:27:7 › Browse panel › @p0e2e User opens Browse panel and searches for views (20.1s)
  ✓  3 browseComponent.test.ts:41:7 › Browse panel › @p0e2e User opens Browse panel and searches for dashboard (20.1s)
  ✓  4 homepage.test.ts:4:7 › HomePage › @smoke HomePage (19.7s)
  -  5 topNav.test.ts:9:8 › Top Navigation Bar › @smoke Top Navigation should be visible (for new Homepage)
  ✘  6 topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu (36.9s)
  1) topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 

    page.goto: Timeout 30000ms exceeded.
    =========================== logs ===========================
    navigating to "https://infra-integration-costa.collibra.com/apps/", waiting until "load"
    ============================================================

       at ../pages/BasePage.ts:18

      16 |   async visit(path: string): Promise<void> {
      17 |     await Promise.all([
    > 18 |       this.page.goto(path),
         |                 ^
      19 |       expect(this.page).toHaveURL(path),
      20 |     ]);
      21 |     await expect(this.topNavComponent.parentContainer).toBeVisible();

        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/BasePage.ts:18:17)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:18:19
        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:17:16)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/tests/topNav.test.ts:5:18

    attachment #1: Console Errors (text/plain) ─────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/logs.txt
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: video (video/webm) ──────────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/video.webm
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #4: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #5: screenshot (image/png) ──────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1) topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 

    page.goto: Timeout 30000ms exceeded.
    =========================== logs ===========================
    navigating to "https://infra-integration-costa.collibra.com/apps/", waiting until "load"
    ============================================================

       at ../pages/BasePage.ts:18

      16 |   async visit(path: string): Promise<void> {
      17 |     await Promise.all([
    > 18 |       this.page.goto(path),
         |                 ^
      19 |       expect(this.page).toHaveURL(path),
      20 |     ]);
      21 |     await expect(this.topNavComponent.parentContainer).toBeVisible();

        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/BasePage.ts:18:17)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:18:19
        at HomePage.visit (/Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/pages/HomePage.ts:17:16)
        at /Users/costa.giannakopoulos/Workspace/e2e-tests/playwright/tests/topNav.test.ts:5:18

    attachment #1: Console Errors (text/plain) ─────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/logs.txt
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #2: video (video/webm) ──────────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/video.webm
    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #4: trace (application/zip) ─────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip
    Usage:

        npx playwright show-trace test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/trace-1.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

    attachment #5: screenshot (image/png) ──────────────────────────────────────────────────────────
    test-assets/topNav-Top-Navigation-Bar-p0e2e-User-navigates-to-glossary-page-using-Products-menu/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 flaky
    topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 
  2 skipped
  1 flaky
    topNav.test.ts:17:7 › Top Navigation Bar › @p0e2e User navigates to glossary page using Products menu 
  2 skipped
  3 passed (2.0m)

[costa-collibra]

@AmsterGet can you reproduce this error on your side?

[AmsterGet]

Hello @costa-collibra ! Seems like I've reproduced the issue. In my case the issue occurs (without any errors in console) while using static .skip() annotations in tests and retries > 0. Do you have static .skip() annotations in your tests?

[costa-collibra]

Hi

Yes indeed we have 2 tests with the fixme annotation which causes them to be skipped.

Awesome work on reproducing that.

This is exactly our use case

[costa-collibra]

Here is one of the tests

test.fixme(
    '@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

[AmsterGet]

@costa-collibra I'm trying to find possible workarounds for this case before a fix is available. So far, I haven't seen In Progress infinite launches when using non-static annotations like:

test('@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      test.fixme();
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

Also I cannot reproduce the issue with playwright/test version lower than 1.31.0.

Could you please recheck these two options on your side and confirm/decline them?

[costa-collibra]

@AmsterGet Interesting.

So you this bug is only reproducible when:

• using playwright version 1.31.0
• test.skip() or test.fixme() is used like this:

test.fixme(
    '@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

If you just add test.fixme() inside the test it works?

My current deployment of report portal is currently down whilst we figure which team is going to take responsibility for it and that I have a meeting this afternoon.

All being well we can have it back tomorrow and I can verify your questions, please bare with me till then :)

[costa-collibra]

@AmsterGet Hi

The results are no longer hanging when i changed the syntax to this:

test('@smoke Top Navigation should be visible (for new Homepage)', async ({
    homePage,
  }) => {
    test.fixme();
    const { topNavComponent } = homePage;
    await topNavComponent.verifyTopNavIsVisible();
  });

This is with retries set to 1 in the Playwright config.

Most people do however use the other syntax.

Will you be looking to fix this? In the meantime at least we have a workaround

[AmsterGet]

Hello @costa-collibra ! I've made some changes to minimize this issue occurrence, but with current agent implementation we cannot fully avoid this issue without providing additional data from Playwright runner to the reporter. I plan to post an issue to the Playwright repo with my investigations and suggestions, I'll refer it here once it be ready. For now the issue reproduced only in case of the hook error and retries enabled that leads to retrying skipped tests marked with .skip() or .fixme() static annotations in the same suite (file or describe block).

I've also added the issue troubleshooting section to the readme file with found workaround and link to this issue.

Feel free to add your feedback on this :)

[AmsterGet]

@costa-collibra I've just released the new version 5.0.9 with some fixes that I've mentioned above. Could you please try and confirm that the issue no longer exists?

[costa-collibra]

@AmsterGet thanks for this update. I will update the library today and feedback.

If you can also tag me or send me a link to the issue when you log it with Playwright that would be great. If i can help with any feedback with them I would certainly be happy to do so

[costa-collibra]

@AmsterGet with your fix in can I go back to this code

test.describe('Top Navigation Bar', () => {
  test.fixme(
    '@smoke Top Navigation should be visible (for new Homepage)',
    async ({ homePage }) => {
      const { topNavComponent } = homePage;
      await topNavComponent.verifyTopNavIsVisible();
    },
  );

or do i need to keep it inside the test

[AmsterGet]

@costa-collibra yes, you can try your previous solution. The issue will be reproduced only for case described in Issues troubleshooting section.

[costa-collibra]

@AmsterGet it worked Thanks alot :)

[sumit-gupta91]

@AmsterGet I am seeing a similar behaviour. Some of it was fixed in latest version 5.0.11 but I think one case is missed and that is still causing Report portal to hang. If the test has test.fail() in it then tests seem to hang on report portal.

[sumit-gupta91]

@AmsterGet In this line - https://github.com/razorpay/frontend-website/pull/2044 Should we also not account for test.fail() static annotations ?

[AmsterGet]

@sumit-gupta91 The page you shared is no longer available. Could you please provide more details?