Requests to domains without DNS entry pile-up across test runs and then block requests in subsequent tests

[gruhn]

What is your Scenario?

Ok, this one was really hard to debug.

I have a scenario where we run tests against our website. The first 5 tests are fine but then suddenly from the 6th test onwards XHR requests on the website are not completing anymore. Everything seems to be loading forever. If I cancel the test run and restart it, I get the same outcome. It seems to have nothing to do with the tests themselves. If I rearrange the order of tests: same outcome. In fact, as you can see in the minimal demo I provided below, I can even run a single test just multiple times in a row and still get the same outcome: The first 5 tests are no problem then from the 6th test onwards they all fail.

For the sake of concreteness consider the demo application provided below. You can click a button; this loads a Chuck Norris joke via API; while waiting for a response we show a "loading..." text; once the response is there, the "loading..." text is replaced with the joke. The corresponding E2E test just clicks on the button and then checks whether the output text is not equal to "loading...".

https://user-images.githubusercontent.com/26570572/183877115-695fe4f7-f5d6-4233-a0f5-233cc139e55c.mov

We just repeat the same test again and again. As you can see, it works out 5 times and then keeps failing from there. Miraculously, this is impossible to reproduce manually. It only happens during the test run.

Here is one thing I haven't told you yet. On page load, the website also performs another request to a domain which has no registered DNS entry:

fetch("https://some-random-domain-without-dns-entry-139481023.com", {
    "headers": { 
        // no idea why, but without this header I can't reproduce the behavior
        "content-type": "application/json"
    }
})

these requests don't just fail. They seem to "pile up" across test runs and at the time of the 6th test, we hit Chromes connections-per-server limit which is 6:

What is the Current behavior?

(Not sure how to separate this from the "What is your Scenario?")

What is the Expected behavior?

(Not sure how to separate this from the "What is your Scenario?")

What is your public website URL? (or attach your complete example)

Check this repository:

https://github.com/gruhn/testcafe-dns-bug-demo

It includes the website and the test code. It also exposes the website via GitHub Pages:

https://gruhn.github.io/testcafe-dns-bug-demo/

You can run the test code with: npm run test but it's a really minimal setup. You probably don't need to pull the repository.

What is your TestCafe test code?

import { Selector } from 'testcafe'

fixture('Bug Demo')
    .page("https://gruhn.github.io/testcafe-dns-bug-demo/");

for (let i = 1; i < 10; i++) {
    test(`Test ${i}`, async t => {
        await t.click("#button")
        await t.expect(Selector("#output").innerText).notEql("loading...")
    })
}

same as in the above linked repository:

https://github.com/gruhn/testcafe-dns-bug-demo/blob/master/test.spec.js

Your complete configuration file

it's just the default configuration

Your complete test report

> test
> testcafe chrome test.spec.js --debug-on-fail

 Running tests in:
 - Chrome 104.0.5112.79 / Monterey 12

 Bug Demo
 ✓ Test 1
 ✓ Test 2
 ✓ Test 3
 ✓ Test 4
 ✓ Test 5

----
Chrome 104.0.5112.79 / macOS 10.15.7
DEBUGGER PAUSE ON FAILED TEST:
AssertionError: expected 'loading...' to not deeply equal 'loading...'

Browser: Chrome 104.0.5112.79 / macOS 10.15.7

5 |
6 |for (let i = 1; i < 10; i++) {
7 |    test(`Test ${i}`, async t => {
8 |        await t.debug()
9 |        await t.click("#button")
> 10 |        await t.expect(Selector("#output").innerText).notEql("loading...")
11 |    })
12 |}

Screenshots

see "What is your Scenario?"

Steps to Reproduce

see "What is your Scenario?"

TestCafe version

1.20.1

Node.js version

v16.5.0

Command-line arguments

testcafe chrome test.spec.js --debug-on-fail

Browser name(s) and version(s)

Chrome 104.0.5112.79

Platform(s) and version(s)

macOS 10.15.7

Other

No response

[miherlosev]

Hi @gruhn

Thank you for the shared information. I've reproduced the issue.

[miherlosev]

Hi @gruhn,

TestCafe runs tests using the URL-rewritten proxy. This approach is good. However, there is a way to improve the stability and speed of test execution - the native browser automation API. We have a test execution mode uses native browser automation - we call it the Proxyless mode. In Proxyless mode, a few issues are already fixed. By the way, this issue was also fixed in Proxyless mode. Try running your tests in Proxyless mode and let us know the results. This option is available in all interfaces:

// Command-line
testcafe chrome tests --experimental-proxyless

// Programmatic
const testcafe = await createTestCafe({ experimentalProxyless: true });

// Configuration file
{
   "experimentalProxyless": "true"
}   

Note that at present it is an experimental mode. Also, the Proxyless mode is implemented only in Google Chrome. It will not work correctly if you run tests in a non-Chrome browser or in a combination of other browsers.

[gruhn]

Hi, thanks for the response. I updated to the latest Testcafe version in my demo repo and added the --experimental-proxyless flag as you suggested:

https://github.com/gruhn/testcafe-dns-bug-demo/commit/6b7d9f1a9a62c3f1928e3522f308ffa03eb48376

But it still reproduces the same behavior.

[miherlosev]

Please upgrade to the latest TestCafe version(2.1.1-alpha.2) and run the tests again.

[gruhn]

That fixes the issue indeed, thanks 🎉

[miherlosev]

I'm happy to hear that.