Failed to carry over a Cookie from the previous session with UserDataAccess=FOR_HERE to the succeeding session with UserDataAccess=TO_GO of a single Profile "Picasso"

[kazurayam]

A test failed; CarryingCookieOverSessionsViaChromeProfileTest # test_carrying_cookie_over_sessions_via_profile

org.junit.ComparisonFailure: expected:<... 24 Mar 2024 22:53:1[2] GMT> but was:<... 24 Mar 2024 22:53:1[4] GMT>
    at org.junit.Assert.assertEquals(Assert.java:115)
    at org.junit.Assert.assertEquals(Assert.java:144)
    at org.junit.Assert$assertEquals.callStatic(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:197)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:217)
    at com.kazurayam.webdriverfactory.chrome.CarryingCookieOverSessionsViaChromeProfileTest.test_carrying_cookie_over_sessions_via_profile(CarryingCookieOverSessionsViaProfileTest.groovy:89)
    ...

[kazurayam]

> Task :compileJava
> Task :compileGroovy NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes
> Task :compileTestJava NO-SOURCE
> Task :compileTestGroovy
> Task :processTestResources NO-SOURCE
> Task :testClasses
[Test worker] INFO io.github.bonigarcia.wdm.WebDriverManager - Using chromedriver 123.0.6312.58 (resolved driver for Chrome 123)
[Test worker] INFO io.github.bonigarcia.wdm.online.Downloader - Downloading https://storage.googleapis.com/chrome-for-testing-public/123.0.6312.58/mac-x64/chromedriver-mac-x64.zip
[Test worker] INFO io.github.bonigarcia.wdm.online.Downloader - Extracting driver from compressed file chromedriver-mac-x64.zip
[Test worker] INFO io.github.bonigarcia.wdm.WebDriverManager - Exporting webdriver.chrome.driver as /Users/kazuakiurayama/.cache/selenium/chromedriver/mac64/123.0.6312.58/chromedriver
Starting ChromeDriver 123.0.6312.58 (6b4b19e9dfbb93aa414dc045bd445287977d8d7a-refs/branch-heads/6312_46@{#3}) on port 46586
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
Mar 25, 2024 9:15:39 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
options => Capabilities {browserName: chrome}
userProfile => {"userProfile":"Picasso","userDataDir":"/Users/kazuakiurayama/Library/Application Support/Google/Chrome","cacheDirectoryName":"Profile 17"}
userDataAccess => FOR_HERE

<<<< Request received
{
"method": "GET",
"uri": "/",
"headers": [
    {"Accept-encoding": ["gzip, deflate, br"]}
    {"Sec-ch-ua": ["\"HeadlessChrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""]},
    {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]},
    {"Sec-fetch-dest": ["document"]},
    {"Sec-fetch-user": ["?1"]},
    {"Connection": ["keep-alive"]},
    {"Host": ["127.0.0.1"]},
    {"Sec-fetch-site": ["none"]},
    {"Sec-ch-ua-platform": ["\"macOS\""]},
    {"Sec-fetch-mode": ["navigate"]},
    {"User-agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/123.0.6312.59 Safari/537.36"]},
    {"Upgrade-insecure-requests": ["1"]},
    {"Sec-ch-ua-mobile": ["?0"]},
],
"body": """"""
}
==== Cookies cooked
in the request:  no cookies found
in the response: cookies=[timestamp=Mon, 25 Mar 2024 00:15:39 GMT; Max-Age=60;]

>>>> Response sent
/ is a directory.
[Test worker] INFO com.kazurayam.webdriverfactory.chrome.ChromeDriverFactoryImpl - copied 399 files from /Users/kazuakiurayama/Library/Application Support/Google/Chrome/Profile 17 into /var/folders/7m/lm7d6nx51kj0kbtnsskz6r3m0000gn/T/__user-data-dir__1374382028392451945/Profile 17
Starting ChromeDriver 123.0.6312.58 (6b4b19e9dfbb93aa414dc045bd445287977d8d7a-refs/branch-heads/6312_46@{#3}) on port 36090
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
Mar 25, 2024 9:15:41 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
options => Capabilities {browserName: chrome}
userProfile => {"userProfile":"Picasso","userDataDir":"/var/folders/7m/lm7d6nx51kj0kbtnsskz6r3m0000gn/T/__user-data-dir__1374382028392451945","cacheDirectoryName":"Profile 17"}
userDataAccess => TO_GO

<<<< Request received
{
"method": "GET",
"uri": "/",
"headers": [
    {"Accept-encoding": ["gzip, deflate, br"]}
    {"Sec-ch-ua": ["\"HeadlessChrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\""]},
    {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"]},
    {"Sec-fetch-dest": ["document"]},
    {"Sec-fetch-user": ["?1"]},
    {"Connection": ["keep-alive"]},
    {"Host": ["127.0.0.1"]},
    {"Sec-fetch-site": ["none"]},
    {"Sec-ch-ua-platform": ["\"macOS\""]},
    {"Sec-fetch-mode": ["navigate"]},
    {"User-agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/123.0.6312.59 Safari/537.36"]},
    {"Upgrade-insecure-requests": ["1"]},
    {"Sec-ch-ua-mobile": ["?0"]},
],
"body": """"""
}
==== Cookies cooked
in the request:  no cookies found
in the response: cookies=[timestamp=Mon, 25 Mar 2024 00:15:41 GMT; Max-Age=60;]

>>>> Response sent
/ is a directory.
timestamp1 => timestamp=Mon, 25 Mar 2024 00:15:39 GMT; expires=Mon, 25 Mar 2024 00:16:39 GMT; path=/; domain=127.0.0.1
timestamp2 => timestamp=Mon, 25 Mar 2024 00:15:41 GMT; expires=Mon, 25 Mar 2024 00:16:41 GMT; path=/; domain=127.0.0.1

Expected :Mon, 25 Mar 2024 00:15:39 GMT
Actual   :Mon, 25 Mar 2024 00:15:41 GMT
<Click to see difference>

org.junit.ComparisonFailure: expected:<..., 25 Mar 2024 00:15:[39] GMT> but was:<..., 25 Mar 2024 00:15:[41] GMT>
    at org.junit.Assert.assertEquals(Assert.java:115)
    at org.junit.Assert.assertEquals(Assert.java:144)
    at org.junit.Assert$assertEquals.callStatic(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:197)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:217)
    at com.kazurayam.webdriverfactory.chrome.CarryingCookieOverSessionsViaChromeProfileTest.test_carrying_cookie_over_sessions_via_profile(CarryingCookieOverSessionsViaProfileTest.groovy:95)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
    at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at jdk.proxy2/jdk.proxy2.$Proxy5.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

> Task :test
com.kazurayam.webdriverfactory.chrome.CarryingCookieOverSessionsViaChromeProfileTest > test_carrying_cookie_over_sessions_via_profile FAILED
    org.junit.ComparisonFailure at CarryingCookieOverSessionsViaProfileTest.groovy:95
1 test completed, 1 failed
> Task :test FAILED
> Task :jacocoTestReport
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///Users/kazuakiurayama/github/chromedriverfactory/build/reports/tests/test/index.html
* Try:
> Run with --scan to get full insights.
BUILD FAILED in 34s
4 actionable tasks: 4 executed

In the above log, I could see

==== Cookies cooked
in the request:  no cookies found
...

This message appears twice! That should not be the case.

At the sencond time, I except a Cookie is sent from Chrome to the Server. But actually not. Why? This is the problem.

[kazurayam]

I should check the folder

"userDataDir":"/var/folders/7m/lm7d6nx51kj0kbtnsskz6r3m0000gn/T/__user-data-dir__1374382028392451945",

to see if it contains a Cookie somewhere, or not.

[kazurayam]

Yes, I found a Cookie file which contains '127.0.0.1timestamp'

[kazurayam]

Then, I wonder why Chrome did not pickup the "timestamp" Cookie from the storage and send to the server at 127.0.0.1 ???

[kazurayam]

I guess it is likely that Chrome has been developed more and more security conscieous. So that Chrome detected the Cookies file on the disk was dirty. So that Chrome ignored it. ..... I am not sure.

If that is the case, this project is no longer valuable at all.

[kazurayam]

52 could be related to this.

e) Later on, after the WebDriver is created, I observe that the Cookies file has been altered. The SQLite database has been emptied. 😲

My question is: How to tell the WebDriver instantiation process not to empty the Cookies?

[kazurayam]

I have a wild guess. See

https://github.com/kazurayam/chromedriverfactory/issues/56

The message "Chrome is controlled by automated software" implies that Chrome would be able to empty the Cookies when it recognized being controlled by automated software.

---

If I do a trick to remove the "Chrome is controlled by automated software", then the Chookies might become working again.

[kazurayam]

Chrome with --headless had a bug that it can not handle cookies properly.

Chrome v109 introduced another option --headless=new which enables us to handle cookies properly.

done at v0.7.2