DevExpress / testcafe

A Node.js tool to automate end-to-end web testing.
https://testcafe.io
MIT License
9.82k stars 671 forks source link

Test browsers hang intermittently when scrolling is present #8208

Open ayemelyanenko-chegg opened 3 months ago

ayemelyanenko-chegg commented 3 months ago

What is your Scenario?

Would like to run tests with scroll event present without having the browsers hang

What is the Current behavior?

When running tests, the tests browsers intermittently hang without any further action and without timing out

If the scrollOccured is commented out (check test code), then there are seemingly no hanging issues

What is the Expected behavior?

Would like to run tests with scroll event present without having the browsers hang without ever exiting

What is the public URL of the test page? (attach your complete example)

https://github.com/ayemelyanenko-chegg/nextjs-nodejs-example

What is your TestCafe test code?

import * as path from "path";
import { ClientFunction, Selector, t } from "testcafe";
export const emailForSignIn = Selector('input[id*="email"]');
export const passForSignin = Selector('input[type="password"]');
export const signInButton = Selector("button").withText("Sign in");
export const getStartedButton = Selector("span").withExactText("Get started");
const upload = Selector('div[data-test="file-upload-area-upload-drop-zone"]');
const fileUploadStatusModal = Selector("#notes-upload-status-modal");
const frame = Selector('iframe[name="CodePen"]');
const btnWithText = Selector("button").withText("Upload a file...");
const uploadOntoCanvas = Selector(
  'input[data-test = "file-upload-area-hidden-input-area"]'
);
const pathToFile = path.resolve(__dirname, `./sample_image.jpeg`);
const nextModal = Selector('div[id="notes-upload-status-modal"]');

export const checkScrollEvent = ClientFunction(() => {
  return new Promise((resolve) => {
    window.addEventListener(
      "scroll",
      () => {
        resolve(true); // Resolve the Promise when scroll event occurs
      },
      { once: true }
    ); // Use { once: true } to remove the listener after the first scroll event
  });
});

// user log in
async function userLogInWithCredentials(
  email: string,
  password: string
): Promise<void> {
  await t.setNativeDialogHandler(() => true);
  await t.navigateTo(
    "https://www.chegg.com/auth?redirect=https%3A%2F%2Fwww.chegg.com%2F"
  );
  await t
    .typeText(emailForSignIn, email, { paste: true })
    .typeText(passForSignin, password, { paste: true })
    .click(signInButton)
    .wait(6000);
}

fixture`Tests should not hang`
  .meta({
    author: "PLA",
    app: "",
  })
  .beforeEach(async (t) => {
    await userLogInWithCredentials("test_user_upload@chegg.us", "Password@1");
    await t.navigateTo("/pla/notesbychegg");
    await t.eval(() => {
      document.addEventListener("DOMContentLoaded", () => {
        const originalOpen = window.open;
        window.open = function (url, target, features) {
          return originalOpen.call(window, url, target, features);
        };
      });
    });
  });

test.meta({
  testID: "t-0001",
})("Tests should not hang", async (t) => {
  await t.click(getStartedButton);
  const scrollOccurred = await checkScrollEvent();
  // Assert that the scroll event occurred. Page should scroll to the upload container
  await t.expect(scrollOccurred).ok("Scroll event did not occur");
  await t.clearUpload(uploadOntoCanvas);

  await t.setFilesToUpload(uploadOntoCanvas, pathToFile);
  await t
    .expect(fileUploadStatusModal.exists)
    .ok()
    .expect(fileUploadStatusModal.visible)
    .ok();
});

test.meta({
  testID: "t-0002",
})("Tests should not hang", async (t) => {
  await t.click(getStartedButton);
  const scrollOccurred = await checkScrollEvent();
  // Assert that the scroll event occurred. Page should scroll to the upload container
  await t.expect(scrollOccurred).ok("Scroll event did not occur");
  await t.clearUpload(uploadOntoCanvas);
  await t.setFilesToUpload(uploadOntoCanvas, pathToFile);
  await t
    .expect(fileUploadStatusModal.exists)
    .ok()
    .expect(fileUploadStatusModal.visible)
    .ok();
});

test.meta({
  testID: "t-0002",
})("Tests should not hang", async (t) => {
  await t.click(getStartedButton);
  const scrollOccurred = await checkScrollEvent();
  // Assert that the scroll event occurred. Page should scroll to the upload container
  await t.expect(scrollOccurred).ok("Scroll event did not occur");
  await t.clearUpload(uploadOntoCanvas);

  await t.setFilesToUpload(uploadOntoCanvas, pathToFile);
  await t
    .expect(fileUploadStatusModal.exists)
    .ok()
    .expect(fileUploadStatusModal.visible)
    .ok();
});

Your complete configuration file

No response

Your complete test report

No response

Screenshots

Tests hang and the console sometimes displays this error related to [r2wc]

Screenshot 2024-06-07 at 3 07 05 PM

Steps to Reproduce

  1. Clone https://github.com/ayemelyanenko-chegg/nextjs-nodejs-example
  2. cd into repo
  3. cd into my-app
  4. yarn install
  5. yarn testcafe chrome tests/sampleUploadTest.ts -c 3 -L
  6. If tests pass and don't hang, exit live mode and then retrigger tests again with the command from step 5. Several attempts like this will eventually cause browsers to hang

TestCafe version

3.6

Node.js version

20.11

Command-line arguments

yarn testcafe chrome tests/sampleUploadTest.ts -c 3 -L

Browser name(s) and version(s)

No response

Platform(s) and version(s)

No response

Other

No response

github-actions[bot] commented 3 months ago

We appreciate you taking the time to share information about this issue. We reproduced the bug and added this ticket to our internal task queue. We'll update this thread once we have news.

Bayheck commented 3 months ago

Hello, We do not recommend using live mode and concurrency in combination.

It is difficult to work on the issue because of how unstable the usage scenario is.

Please try running your tests without concurrency and see if the tests fail.

ayemelyanenko-chegg commented 3 months ago

@Bayheck Hello, the live mode with concurrency was an example so that the issue is easily reproducible is visible. This also happens without the live mode as our tests run in GitLab without live mode. If the same tests are run with just concurrency but scrolling is present, there is a chance that browsers will hang. We also tried running without concurrency but with scrolling present and the browsers hang intermittently still.

Bayheck commented 2 months ago

Hello,

I tried to run your example without concurrency and live mode and got browser hanging on 1 out of 10-11 runs. How often do you get this behavior? Is issue occurrence frequency different on your CI and local runs?

ayemelyanenko-chegg commented 2 months ago

Hello, it was a fairly common occurrence because we tended to run those tests quite often and in multiple PR instance. The issue with this is that this behavior causes browsers to freeze so the entire suite would time out and that would cause a hassle. At this point, we are alright because we no longer do the scroll as we commented out that portion of the tests.