DevExpress / testcafe

A Node.js tool to automate end-to-end web testing.
https://testcafe.io
MIT License
9.82k stars 671 forks source link

Request Hook on skipped Test affects next Test (native automation) #8229

Closed htho closed 1 month ago

htho commented 3 months ago

What is your Scenario?

I have a skipped test, which uses test.requestHooks(myHook)

What is the Current behavior?

The test is skipped, but the hook is applied to the next test, that is not skipped.

What is the Expected behavior?

The hook should not be applied to the next test.

What is the public URL of the test page? (attach your complete example)

https://devexpress.github.io/testcafe/example

What is your TestCafe test code?

import { fixture, RequestMock, Selector, test } from "testcafe";

fixture("Getting Started")
    .page("https://devexpress.github.io/testcafe/example");

test("before the skipped", async t => {
    await t.expect(Selector("#main-form").exists).ok();
});

test.skip.requestHooks(
    RequestMock().onRequestTo(/.*example.*/).respond("", 404)
)("skipped", async t => {
    // nothing to do because it is skipped
});

test("after the skipped", async t => {
    await t.expect(Selector("#main-form").exists).ok(); // <-- FAILS
});

test("after after the skipped", async t => {
    await t.expect(Selector("#main-form").exists).ok();
});

Your complete configuration file

No response

Your complete test report

C:\Dev\github.com\htho\tc> .\node_modules\.bin\testcafe chrome .\test.tc.ts
 Running tests in:
 - Chrome 126.0.0.0 / Windows 10

 Getting Started
 √ before the skipped
 - skipped
 × after the skipped

   1) AssertionError: expected false to be truthy

      Browser: Chrome 126.0.0.0 / Windows 10

         10 |    RequestMock().onRequestTo(/.*example.*/).respond("", 404)
         11 |)('skipped', async t => {
         12 |    // nothing to do because it is skipped
         13 |});
         14 |test('after the skipped', async t => {
       > 15 |    await t.expect(Selector("#main-form").exists).ok();
         16 |});
         at <anonymous> (C:\Dev\github.com\htho\tc\test.tc.ts:15:51)
         at <anonymous> (C:\Dev\github.com\htho\tc\test.tc.ts:8:71)
         at __awaiter (C:\Dev\github.com\htho\tc\test.tc.ts:4:12)
         at <anonymous> (C:\Dev\github.com\htho\tc\test.tc.ts:14:37)

 1/2 failed (5s)
 1 skipped
C:\Dev\github.com\htho\tc> .\node_modules\.bin\testcafe chrome .\test.tc.ts
 Running tests in:
 - Chrome 126.0.0.0 / Windows 10

 Getting Started
 √ before the skipped
 - skipped
 × after the skipped

   1) AssertionError: expected false to be truthy

      Browser: Chrome 126.0.0.0 / Windows 10

         13 |)("skipped", async t => {
         14 |    // nothing to do because it is skipped
         15 |});
         16 |
         17 |test("after the skipped", async t => {
       > 18 |    await t.expect(Selector("#main-form").exists).ok();
         19 |});
         20 |
         21 |test("after after the skipped", async t => {
         22 |    await t.expect(Selector("#main-form").exists).ok();
         23 |});

         at <anonymous> (C:\Dev\github.com\htho\tc\test.tc.ts:18:51)
         at <anonymous> (C:\Dev\github.com\htho\tc\test.tc.ts:8:71)
         at __awaiter (C:\Dev\github.com\htho\tc\test.tc.ts:4:12)
         at <anonymous> (C:\Dev\github.com\htho\tc\test.tc.ts:17:37)

 √ after after the skipped

 1/3 failed (5s)
 1 skipped

Screenshots

No response

Steps to Reproduce

  1. Create test.tc.ts with the content from above
  2. Run testcafe chrome test.tc.ts

TestCafe version

3.6.2

Node.js version

v20.11.0

Command-line arguments

chrome test.tc.ts

Browser name(s) and version(s)

Chrome 126

Platform(s) and version(s)

Windows 10

Other

This only happens in native automation. Without it, the tests run fine: chrome .\test.tc.ts --disable-native-automation

github-actions[bot] commented 3 months ago

We appreciate you taking the time to share information about this issue. We reproduced the bug and added this ticket to our internal task queue. We'll update this thread once we have news.