"Error: Can only get response body on requests captured after headers received" when running test on local environment and with experimental proxyless mode

[ayemelyanenko-chegg]

What is your Scenario?

Run a test against a local environment and with experimental proxyless mode

What is the Current behavior?

There is an error:

Error: Can only get response body on requests captured after headers received.

In the screenshot, a blank page is seen

What is the Expected behavior?

The test should pass in experimental proxyless mode as it passes without the experimental proxyless mode

What is your public website URL? (or attach your complete example)

As this is an internal site and works against a reverse-proxy and localhost, steps to reproduce would not work here.

Attaching testcafe.log instead in the test report section

What is your TestCafe test code?

fixture`flagging testcases`
  .meta({
    fixtureID: 'f-0001',
    author: 'GDG',
    app: '',
  })
  .beforeEach(async (t) => {
    email = await sharedHelpers.createCSSubUser();
    console.log('Logging in with ' + email);
    await sharedHelpers.userLogInWithCredentials(email, sharedHelpers.password);
    await t.navigateTo(
      `${config.baseUrl}` +
        '/homework-help/questions-and-answers/east-university-eu-754-students-enrolled-math-1324-math-1325-535-enrolled-math-1324-257-en-q21691469?trackid=5fd398a8c83d&strackid=07caa782f378'
    );
  });

test.meta({
  testID: 't-0001',
  testType: 'Smoke',
  testRailCaseId: '277066',
  priority: 'p1',
})('Validate the flagging icon is present on the tbs page', async (t) => {
  // sometimes tests fail due to not setting the native dialog handler, something to do with qa page
  console.log('Starting flagging component validation');
  await t
    .expect(locators.flagButton.visible)
    .ok()
    .click(locators.flagButton)
    .expect(locators.flagDialogueBox.visible)
    .ok()
    .expect(locators.flagDialogueBox.visible)
    .ok()
    .expect(locators.flagReasonInappropriate.visible)
    .ok()
    .expect(locators.flagReasonSpam.visible)
    .ok()
    .expect(locators.flagReasonOther.visible)
    .ok()
    .expect(locators.flagTextBox.visible)
    .ok();
});

Your complete configuration file

module.exports = {
  screenshots: {
    takeOnFails: true,
    fullpage: true,
    pathPattern:
      '${FIXTURE}_${USERAGENT}/${TEST}/${DATE}_${TIME}__${QUARANTINE_ATTEMPT}/${FILE_INDEX}.png',
    path: './testcafe/screenshots',
  },
  clientScripts: [{ module: 'axe-core/axe.min.js' }],
  reporter: [
    {
      name: 'spec-time',
    },
    {
      name: 'slack-errors-only',
    },
    {
      name: 'list',
      output: 'testcafe/reports/report.txt',
    },
    {
      name: 'xunit',
      output: 'testcafe/reports/xunit.xml',
    },
    {
      name: 'html',
      output: 'testcafe/reports/report.html',
    },
  ],
  quarantineMode: false,
  skipJsErrors: true,
  skipUncaughtErrors: true,
  experimentalProxyless: true,
  selectorTimeout: 15000,
  assertionTimeout: 15000,
  pageLoadTimeout: 20000,
  speed: 1,
  qrCode: true,
  color: true,
  cache: true,
  stopOnFirstFail: false,
  // eslint-disable-next-line max-params
  filter: function (_, _, _, testMeta, fixtureMeta) {
    const environment = process.env.CHEGG_ENV || 'test';

    if (fixtureMeta.env) {
      if (Array.isArray(fixtureMeta.env)) {
        return fixtureMeta.env.includes(environment);
      }

      return fixtureMeta.env === environment;
    }

    if (!testMeta.env) return true;

    if (Array.isArray(testMeta.env)) {
      return testMeta.env.includes(environment);
    }

    return testMeta.env === environment;
  },
};

Your complete test report

testcafe.log

Screenshots

Steps to Reproduce

As this is an internal site and works against a reverse-proxy and localhost, steps to reproduce would not work here.

Attaching testcafe.log instead in the test report section

TestCafe version

2.3.1

Node.js version

16.17.1

Command-line arguments

export DEBUG="testcafe:,hammerhead:" && yarn testcafe chrome testcafe/tests/flaggingTest.ts --experimental-proxyless 2> testcafe.log

Browser name(s) and version(s)

Chrome 109.0.5414.119

Platform(s) and version(s)

On either Mac when running locally or Linux when running on GitLab

Other

No response

[miherlosev]

Hi @ayemelyanenko-chegg,

I investigated the log and found the route that causes an unhandled error. Could you please share more information about this route (statusCode, isRedirect, body length, type)?

[ayemelyanenko-chegg]

Thanks for the response @miherlosev. Could you please recommend a way on how to share more info about the request? I tried to run the test with live mode so that I can open the request in Chrome devtools network tab once the test fails and save a HAR file for you but unfortunately was not able to because of this: https://github.com/DevExpress/testcafe/issues/7514

Tried adding in logger to the test and it showed as undefined after I added it in, maybe I'm adding it in the wrong place?

import { RequestLogger } from 'testcafe';
const logger = RequestLogger();

fixture`flagging testcases`
  .meta({
    fixtureID: 'f-0001',
    author: 'GDG',
    app: '',
  })
  .beforeEach(async (t) => {
    console.log('Validate the flagging component for Chegg subscriber');
    email = await sharedHelpers.createCSSubUser();
    console.log('Logging in with ' + email);
    await sharedHelpers.userLogInWithCredentials(email, sharedHelpers.password);
    await t.navigateTo(
      `https://local.web.test.cheggnet.com` +
        '/homework-help/questions-and-answers/east-university-eu-754-students-enrolled-math-1324-math-1325-535-enrolled-math-1324-257-en-q21691469?trackid=5fd398a8c83d&strackid=07caa782f378'
    );
    await t.addRequestHooks(logger);
            console.log(logger.requests.map(r => r.request.url));
  })

  Failed to load the page at "https://local.web.test.cheggnet.com/homework-help/questions-and-answers/east-university-eu-754-students-enrolled-math-1324-math-1325-535-enrolled-math-1324-257-en-q21691469?trackid=5fd398a8c83d&strackid=07caa782f378".
  Increase the value of the "pageRequestTimeout" variable, enable the "retryTestPages" option, or use quarantine mode to perform additional attempts to execute this test.
  You can find troubleshooting information for this issue at "https://go.devexpress.com/TestCafe_FAQ_ARequestHasFailed.aspx".

  Error details:
  Error: Can only get response body on requests captured after headers received.

  Browser: Chrome 110.0.0.0 / macOS 10.15.7
  Screenshot: /Users/ayemelyanenko/chegg-web/testcafe/screenshots/flagging testcases_Chrome_110.0.0.0_macOS_10.15.7/Validate the flagging icon is present on the tbs page/2023-02-13_15-29-19__1/errors/1.png

     14 |  .beforeEach(async (t) => {
     15 |    console.log('Validate the flagging component for Chegg subscriber');
     16 |    email = await sharedHelpers.createCSSubUser();
     17 |    console.log('Logging in with ' + email);
     18 |    await sharedHelpers.userLogInWithCredentials(email, sharedHelpers.password);
   > 19 |    await t.navigateTo(
     20 |      `https://local.web.test.cheggnet.com` +
     21 |        '/homework-help/questions-and-answers/east-university-eu-754-students-enrolled-math-1324-math-1325-535-enrolled-math-1324-257-en-q21691469?trackid=5fd398a8c83d&strackid=07caa782f378'
     22 |    );
     23 |    await t.addRequestHooks(logger);
     24 |            console.log(logger.requests.map(r => r.request.url));

     at <anonymous> (/Users/ayemelyanenko/chegg-web/testcafe/tests/flaggingTest.ts:19:13)
     at fulfilled (/Users/ayemelyanenko/chegg-web/testcafe/tests/flaggingTest.ts:5:58)```
Unable to send a message to slack
undefined
`

[Artem-Babich]

Hello,

You need to add a request logger before making a request. Please also note that the URL you're using in the test code doesn't match the URL in the log file. It contains a double-slash for some reason. I assume there is a mistake in your routing code.

You can also try finding more info in your router code, if this page is a part of your application.

[ayemelyanenko-chegg]

Hello, I tried setting it before the request as you suggested but it was undefined still. When I try it with any other non local site, it logs fine.

As far as the double slash goes, it's this way because we're passing in an environment variable that contains a slash and then appending part of the url which also contains a slash. Someone appended the extra slash by mistake in the test. I tried it without the double slash and the result was the same.

Nonetheless, I did manage to save a HAR file of the page request while running the test locally with the devtools open. Attaching it here as a zip file and hoping that it helps.

localrequestfailure.har.zip

[Artem-Babich]

Hello,

The har file you sent is broken, it won't open. Please take a screenshot of the required request using the following steps:

1. Open the devtools(F12).
2. Click on the required request.
3. Take a screenshot of the screen.

It should be similar to the following image:

[ayemelyanenko-chegg]

Hello, I can't take a screenshot because live mode doesn't work with experimental proxyless and the devtools panel closes as soon as the browser closes.

Ok if I share the output of the HAR file by another method below?

https://upload.disroot.org/r/9TEoAqdy#PE9OB5sCwxnNMlq9i2GP6ofiAlPjwNnRlWbhvGSCes0=

[Artem-Babich]

Hello,

We are aware of the issue with the live mode in proxyless. However, you don't need to use it to get info about this request. Please open the browser and reproduce test actions manually (without TestCafe), and then export the har file.

[ayemelyanenko-chegg]

Hello.

The thing is, this appears to be a TestCafe issue when using experimental proxyless mode so the HAR file that I provided is actually from a TestCafe test run using experimental proxyless mode but without the live mode. Is it not acceptable in this case?

When I run a test on the local site using TestCafe **without proxyless mode, it works fine.**

When I go to the local site manually, it loads fine too.

[Artem-Babich]

Hello,

To determine the cause of this issue, we need to know the parameters of the request that causes the proxyless to fail. That is why I asked you to reproduce the steps without TestCafe and share the har file containing this request. Once we know the request parameters, we can reproduce the issue locally in proxyless mode.

[ayemelyanenko-chegg]

Hello, apologies. I misunderstood your request but think I got it now.

Below is the har file of the request of the same url without TestCafe.

Hope that it helps, thanks.

https://upload.disroot.org/r/rDeRALkc#k7I2tlTEOA1rrxyseKC1eeCXLJlozHwbILWSuXV/QQI=

[miherlosev]

Hi @ayemelyanenko-chegg,

Thank you for sharing the request information. I investigated it and unfortunately, it's not sufficient to find the cause of the issue. Could you please create an example where we can reproduce the issue locally?

[ayemelyanenko-chegg]

Hi @miherlosev, thanks for the response. I seemed to have reproduced the issue in a repo with a sample app.

Please follow the steps below to set up a sample app. If you have more questions, please reach out to me again.

Thanks.

Steps:

1. Clone https://github.com/ayemelyanenko-chegg/nextjs-nodejs-example

2. CD into nextjs-nodejs-example/

3. Install server side dependencies and start by doing:

   cd api
   npm install
   npm run dev

4. Install react dependencies and start by doing:

   cd my-app
   npm install
   npm run dev

5. In a new terminal window, set up proxy by doing

   npm i -g local-ssl-proxy
   local-ssl-proxy --source 3001 --target 3000

6. Steps 3 to 5 should look like this screenshot

7. Open https://localhost:3001 and type ‘thisisunsafe’ in the browser in order to load the site and bypass security warnings https://cybercafe.dev/thisisunsafe-bypassing-chrome-security-warnings/

Make sure that the sample app loads and it will look like the below screenshot

8. In a new terminal window, CD into nextjs-nodejs-example/my-app and run a sample test by doing: npx testcafe chrome tests/sampleTest.ts --experimental-proxyless

9. Observe the error

10. Note that running the same test without using experimental proxyless will pass the test, see screenshot

[Artem-Babich]

Hi,

Thank you for your sample. I was able to reproduce this issue. We'll research it.

[miherlosev]

Hi @ayemelyanenko-chegg,

The browser interprets the connection to the tested page as not private (because you don't use a valid SSL certificate). In regular mode, TestCafe runs the test over the HTTP protocol, and this issue doesn't happen. In proxyless mode, TestCafe more strictly imitates the native browser behavior and repeats the SSL error. To fix this, you need to add the --ignore-certificate-errors flag to the browser argument.

npx testcafe "chrome --ignore-certificate-errors" tests/sampleTest.ts --experimental-proxyless 

Also, in this PR, I improved the error message displayed in the case of an invalid certificate.

[ayemelyanenko-chegg]

@miherlosev Hi, thank you for the response and the explanation. I will add the flag to the browser parameter as you suggested.