seleniumbase / SeleniumBase

📊 Python's all-in-one framework for web crawling, scraping, testing, and reporting. Supports pytest. UC Mode provides stealth. Includes many tools.
https://seleniumbase.io
MIT License
5.05k stars 945 forks source link

How to Capture Requests with sb in SeleniumBase #3081

Closed xiongzhenglong closed 2 weeks ago

xiongzhenglong commented 2 weeks ago

Description` I'm encountering a challenge when using SeleniumBase. When using the sb instance, I can easily load cookies (e.g., sb.load_cookies(name="cookies.txt")), which is very convenient for maintaining session state. However, I have found that sb cannot directly capture network requests.

On the other hand, when using a driver instance, I can capture network requests, but it’s not straightforward to load cookies or maintain the session state. This creates difficulties when working with websites that require a logged-in state.

Problem The sb instance allows easy cookie loading but doesn't support direct request capture. The driver instance supports request capture but lacks the convenience of loading cookies or maintaining a session state effectively.

this is my code:

from seleniumbase import SB

# Function to intercept and print specific requests
def print_specific_requests(driver):
    for request in driver.requests:        
        print(f"Request URL: {request.url}")
        print(f"Response Status Code: {request.response.status_code}")

def load_cookies_and_intercept():
    # Using `SB` with `wire=True` to enable request interception
    with SB(uc=True, wire=True) as sb:
        # Load cookies from file
        sb.open("https://xxx.io")
        sb.load_cookies(name="xxx.txt")

        # List of URLs to visit
        urls = [
            "https://xxxx",
            "https://yyyy",
        ]

        # Visit each URL and intercept the requests
        for url in urls:
            sb.open(url)
            print_specific_requests(sb.driver)

# Run the function to load cookies and intercept requests
load_cookies_and_intercept()
mdmintz commented 2 weeks ago

Duplicate of https://github.com/seleniumbase/SeleniumBase/issues/1774#issuecomment-1444399269, https://github.com/seleniumbase/SeleniumBase/issues/2032#issuecomment-1693931853, https://github.com/seleniumbase/SeleniumBase/issues/2151#issuecomment-1741234068, etc.

Due to unresolved issues with selenium-wire: https://github.com/wkeeling/selenium-wire/search?q=undetected&type=issues, UC Mode and Wire Mode cannot be combined. Pick one or the other.

(driver.requests is part of selenium-wire)

You can use sb.driver to access the driver from from sb.


There are ways of getting that data in UC Mode without Wire Mode. Examples: