unclecode / crawl4ai

🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper
Apache License 2.0
16.58k stars 1.23k forks source link

Cookies from on_browser_created hook are not preserved #270

Closed flamedmg closed 1 day ago

flamedmg commented 1 week ago

I tried this example and i do not see cookie is passed to the new context created after that hook is executed. Same happens to cookies assigned by the web page or server.


    print("[HOOK] on_browser_created")
    # Example customization: set browser viewport size
    context = await browser.new_context(viewport={'width': 1920, 'height': 1080})
    page = await context.new_page()

    # Example customization: logging in to a hypothetical website
    await page.goto('https://example.com/login')
    await page.fill('input[name="username"]', 'testuser')
    await page.fill('input[name="password"]', 'password123')
    await page.click('button[type="submit"]')
    await page.wait_for_selector('#welcome')

    # Add a custom cookie
    await context.add_cookies([{'name': 'test_cookie', 'value': 'cookie_value', 'url': 'https://example.com'}])

    await page.close()
    await context.close()```
arlse commented 1 week ago

maybe can help link

unclecode commented 5 days ago

@flamedmg @arlse In the new version 0.3.74, you can pass cookies to the constructor directly. This is the way you're going to pass. I plan to release this version by tonight.

cookies = [
    {"name": "sessionId", "value": "abc123", "domain": ".example.com"},
    {"name": "userId", "value": "12345", "domain": ".example.com"}
]
async with AsyncWebCrawler(cookies=cookies) as crawler:
    # rest of code
flamedmg commented 1 day ago

was able to use this approach. but doc still is not correct and confusing. Please fix the documentation and close the issue