[Feat] Remote Playwright Instances - Githubissues

mendableai / firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

https://firecrawl.dev

GNU Affero General Public License v3.0

18.99k stars 1.46k forks source link

[Feat] Remote Playwright Instances #737

Open mschfh opened 1 month ago

mschfh commented 1 month ago

Problem Description Firecrawl lacks support for connecting to remote playwright instances via CDP.

Proposed Feature Implement support for remote Playwright connections over CDP (Chrome DevTools Protocol) to external browser instances.

Implementation Suggestions

add an env var to configure a CDP endpoint
use the CDP endpoint instead of launching a local browser

Use Case

Use Bright Data scraping browser or Zenrows to circumvent captchas, bot detection and other restrictions.

Additional Context

Bright Data's scraping browser: https://docs.brightdata.com/scraping-automation/scraping-browser/code-examples
Zenrows scraping browser: https://www.zenrows.com/solutions/scraping-browser
Playwright's connect_over_cdp: https://playwright.dev/python/docs/api/class-browsertype#browser-type-connect-over-cdp

mschfh commented 1 month ago

@nickscamara Would you accept a PR for playwright-service that uses playwright.chromium.connect_over_cdp() instead of playwright.chromium.launch() if REMOTE_CDP_URL is configured?