Add Proxy Support for Self-Hosted Firecrawl

[techUdayMungalpara]

Add Proxy Support for Self-Hosted Firecrawl

Problem

• Self-hosted Firecrawl can't use proxy services
• Challenges:
  1. IP-based access restrictions
  2. Cloudflare blocking
  3. CAPTCHA interruptions

Proposed Solution

• Implement proxy support using API key from providers (e.g., ZenRow,ScrapingBee)

Implementation

1. Add ProxyConfig class
2. Update config file:

   proxy:
    enabled: true
    provider: "zenrow"
    api_key: "your_api_key_here"

3. Create ProxyClient class for API authentication
4. Modify HTTP client to use ProxyClient
5. Add error handling and logging
6. Write unit tests

Benefits

• Bypass IP restrictions
• Handle Cloudflare protection
• Reduce CAPTCHA interruptions
• Simplify proxy management

Future Enhancements

• Support any proxy providers
• Automatic provider fallback

[mschfh]

Zenrows/Bright Data offer a scraping browser that handles not just proxy rotation but also Cloudflare/CAPTCHA bypass, etc

I opened a feature request to implement support for remote Playwright instances so those can be used.