Closed noahpc closed 1 year ago
TestCafe uses its own cookie processing mechanism. Cookies are stored on the proxy server side. So, cookies which you've seen in the Network tab of DevTools are our service cookies for the synchronization between server and client.
@LavrovArtem - The result I am seeing from this is that browser treats these cookies differently when under test and prevents us from being able to fully test scenarios that involve the browser blocking what should be 3rd party cookies.
Is this expected?
In fact, the more I look, I think that is actually the problem. The client-side testcafe code is properly respecting the domain value; however, it is not realizing that cookies should not be permitted to be set on that domain.
@noahpc
Hello,
It would be great if you could provide us with a project (or a public URL) to show how the TestCafe cookie processing mechanism negatively affects your page application logic. Our team will research it and check for a suitable solution.
@Farfurix - Thank you for the response. Much of my work has been in a private environment, but I have gotten approval to put up a test site that will demonstrate the problem more fully.
I will use this site to show that on Safari with third-party cookies disabled, the third-party cookies are still being set by the test-cafe framework. I'm out of office today, but will update this ticket with the site link in the next couple days. Thanks again for your time!
@Farfurix - I have set up a test page along and hopefully explained the issues I am seeing more clearly. While I setting it up, I realized that the behavior is different (but still unexpected) when a request logger is present versus when there is no request logger. I've added notes about both cases below. Let me know if anything else would be helpful for you. Thanks again!
Explanation of what the test page does: 1) Makes a call to bcp.crwdcntrl.net that will use set-cookie headers to set a 3rd party cookie 2) Pulls in an iframe that is hosted on tags.crwdcntrl.net that is able to determine whether the 3rd party cookie was set 3) Makes a postMessage to the iframe asking whether the 3rd party cookie was able to be set 4) Prints out 1st party if cookie was not set and 3rd party if cookie was set
Expected Behavior: 1) On Chrome, prints out 3rd Party because 3rd party cookies can be set 2) On Safari, prints out 1st Party because 3rd party cookies CANNOT be set
To See Expected Behavior: 1) Simply go directly to http://lotame-public-demo-dev.s3-website-us-east-1.amazonaws.com/ in chrome, observe it says 3rd Party 2) Simply go directly to http://lotame-public-demo-dev.s3-website-us-east-1.amazonaws.com/ in safari, observe it says 1st Party
Run the following tests cases:
Test Code #1: Test Cookies With Logging
import { Selector, RequestLogger, ClientFunction } from 'testcafe';
let requestLogger = RequestLogger(new RegExp('.*'), {
logRequestHeaders: true,
logResponseHeaders: true,
logRequestBody: true,
logResponseBody: true,
stringifyRequestBody: true
});
fixture`Test Cookies With Logging`
.page('http://lotame-public-demo-dev.s3-website-us-east-1.amazonaws.com/')
.requestHooks(requestLogger);
const getUA = ClientFunction(() => navigator.userAgent);
test('The cookies should be set properly', async t => {
const ua = await getUA();
if (ua.includes('Chrome')) {
await t.expect(Selector('#pid').textContent).contains('3rd Party');
} else {
await t.expect(Selector('#pid').textContent).contains('1st Party');
}
});
Test Code #2: Test Cookies Without Logging
import { Selector, ClientFunction } from 'testcafe';
fixture`Test Cookies Without Logging`
.page('http://lotame-public-demo-dev.s3-website-us-east-1.amazonaws.com/');
const getUA = ClientFunction(() => navigator.userAgent);
test('The cookies should be set properly', async t => {
const ua = await getUA();
if (ua.includes('Chrome')) {
await t.expect(Selector('#pid').textContent).contains('3rd Party');
} else {
await t.expect(Selector('#pid').textContent).contains('1st Party');
}
});
Fixture: Test Cookies With Logging Browser: chrome Result: PASS
Fixture: Test Cookies With Logging Browser: safari Result: FAIL
Fixture: Test Cookies Without Logging Browser: chrome Result: FAIL
Fixture: Test Cookies Without Logging Browser: safari Result: PASS
Observations of why I believe the tests are failing:
With Logging Enabled:
Without Logging Enabled:
Hi @noahpc,
Thank you for the project and your explanation. We'll research this behavior and update this thread once we have any results.
Hi, just checking in to see whether there has been any progress on this or whether there is a workaround I can use in the meantime. Thanks!
@noahpc
Hello,
We are still researching this issue. We will update this thread if we have any new information.
@Farfurix - Just checking in here - is there anything we can do to help or any expectation on when this ticket will change state? We are currently having to do a fair bit of our testing manually because of this issue.
Hi @noahpc
The information you provided is enough research the problem. Please wait until we find the cause of the problem.
Just checking in again to see whether anyone was able to research this at all. If not, what would be the timeline for that?
If it is unlikely that you all would have the time to look into it, would it be possible for my team to suggest a fix through PR?
Up to this point, my team hasn't had the time to dig into the codebase , but if we were able to dedicate the time to it and find a potential solution, would you all be open to helping us merge that into the main branch?
Currently, we cannot provide any estimates on this issue. However, your PR would be greatly appreciated.
I am going start looking into submitting a fix PR for this. First step is to find out why the tests don't all pass on a fresh clone and initial npm install.
To close the loop here, I am no longer attempting a fix
Hi @markaconrad
TestCafe runs tests using the URL-rewritten proxy.
This approach is good. However, there is a way to improve the stability and speed of test execution - the native browser automation API.
We have a test execution mode uses native browser automation - we call it the Proxyless
mode.
In Proxyless
mode, a few issues are already fixed.
By the way, this issue was also fixed in Proxyless
mode.
Try running your tests in Proxyless
mode and let us know the results.
This option is available in all interfaces:
// Command-line
testcafe chrome tests --experimental-proxyless
// Programmatic
const testcafe = await createTestCafe({ experimentalProxyless: true });
// Configuration file
{
"experimentalProxyless": "true"
}
Note that at present it is an experimental mode.
Also, the Proxyless
mode is implemented only in Google Chrome. It will not work correctly if you run tests in a non-Chrome browser or in a combination of other browsers.
Hi @noahpc,
This issue is not reproduced with combination of testcafe@3.0.1
and the Google Chrome browser. Feel free to reopen this issue if you encounter it in other browsers.
@miherlosev - thank you for the note. One question though, how would I be able to specify that 3rd party cookies are not supported since by default chrome does support 3rd party cookies and there is no way to specify a custom browser configuration with native automation. (happened to come across another scenario where this would be helpful)
@noahpc
You can run Chrome with --test-third-party-cookie-phaseout
to disable third-party cookies as follows:
npx testcafe "chrome --test-third-party-cookie-phaseout" test.js
Please also note that third-party cookies will be disabled in 2024: https://developer.chrome.com/en/blog/cookie-countdown-2023oct/
What is your Test Scenario?
I am making an http call that has the following response:
What is the Current behavior?
hammerhead is rewriting the response as
What is the Expected behavior?
I would expect the domain to be the same as the original response. TestCafe appears to be removing the domain of the cookies. This results in all cookies looking like first party cookies even when they are 3rd party.
Test Code
Steps to Reproduce:
Your Environment details: