seleniumbase / SeleniumBase

📊 Blazing fast Python framework for web crawling, scraping, testing, and reporting. Supports pytest. Stealth abilities: UC Mode and CDP Mode.
https://seleniumbase.io
MIT License
5.44k stars 983 forks source link

method save_cookies don't work #3218

Closed schamen closed 1 month ago

schamen commented 1 month ago

I don't understand, maybe it's just me, but for some reason sb.save_cookies(name="/usr/src/cookies.txt") doesn't work - the cookie file is not being created. Perhaps this method depends on some module, but all dependencies were installed via install.sh, and the latest version of SeleniumBase 4.32.1 is also installed. Also installed py 3.12

schamen commented 1 month ago

and also why i cant add cookie different site when https://123, with this method. I always get selenium.common.exceptions.InvalidCookieDomainException: Message: invalid cookie domain: Cookie 'domain' mismatch (Session info: chrome=122.0.6261.128) but such a method works in Node.js - await page.setCookie({...}) sb.add_cookie({ 'name': 'name', 'value': 'val', 'path': '/', 'domain': 'oauth.facebook', - for example 'expires': expires, 'SameSite': 'none', 'HostOnly': True, 'secure': True, 'httponly': True, })

schamen commented 1 month ago

I think the driver is incorrectly handling 'SameSite': 'None'

mdmintz commented 1 month ago

From the docs: save_cookies --> Saves the page cookies to the "saved_cookies" folder. You pick the filename, not the folder. If you want more control, you can save the cookies directly with driver.get_cookies(), and then save them where you want them.

Selenium only allows same-origin cookie-loading. If you want to load cookies from one origin to another, then you may need to modify cookies first. (This could change later.)

See the example: SeleniumBase/examples/raw_cookies.py

schamen commented 1 month ago

From the docs: save_cookies --> Saves the page cookies to the "saved_cookies" folder. Thank you for the clarification. I have understood how to set cookies. The main oversight is that there is no possibility to set cookies for a different domain. When can we expect an update that allows setting samesite = none? Is there a way to bypass this limitation at the moment? Or how can I ensure that all session parameters are saved by the browser so that they can be restored later? Thank you

mdmintz commented 1 month ago

I'll modify the domain in the next version of SeleniumBase so that cookies can be loaded into any site from any origin.

mdmintz commented 1 month ago

For cross-origin cookies, try the new version: 4.32.2. (Using the load_cookies() method)

schamen commented 1 month ago

ok. and how get all headers after requset? but after sb.uc_open_with_reconnect

schamen commented 1 month ago

i try set add cookie after sb.uc_open_with_reconnect and before but always get selenium.common.exceptions.InvalidCookieDomainException: Message: invalid cookie domain: Cookie 'domain' mismatch (Session info: chrome=122.0.6261.128)

mdmintz commented 1 month ago

ok. and how get all headers after requset? but after sb.uc_open_with_reconnect

https://github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_cdp_logging.py

from rich.pretty import pprint
from seleniumbase import Driver

driver = Driver(uc=True, log_cdp=True)
try:
    url = "seleniumbase.io/apps/turnstile"
    driver.uc_open_with_reconnect(url, 2)
    driver.uc_gui_handle_captcha()
    driver.sleep(3)
    pprint(driver.get_log("performance"))
finally:
    driver.quit()
mdmintz commented 1 month ago

i try set add cookie after sb.uc_open_with_reconnect and before but always get selenium.common.exceptions.InvalidCookieDomainException: Message: invalid cookie domain: Cookie 'domain' mismatch (Session info: chrome=122.0.6261.128)

Need to use the load_cookies() method with version 4.32.2 (or newer), or manually modify the cookies to change the origins to match the site you want to load the cookies on.

schamen commented 1 month ago

Need to use the load_cookies() method with version 4.32.2 (or newer), or manually modify the cookies to change the origins to match the site you want to load the cookies on.

But why do we need to load cookies from a file when there's the sb.add_cookie method? Or should both methods work.

schamen commented 1 month ago

sorry for offtopic, but how resolve DevTools remote debugging is disallowed by the system admin. under wind10?

schamen commented 1 month ago

I have written all the necessary parameters in cookie.txt, but after the cookies are loaded using the load_cookies method, the domain changes to the domain of the open page after sb.uc_open_with_reconnect. Cookies from another domain must be set for OAuth authorization before or after solving the CF challenge, What i must to do next?

mdmintz commented 1 month ago

Try with 4.32.3, which is now available.