Question about using proxy in WebCrawler

unclecode / crawl4ai

🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper

Apache License 2.0

2.02k stars 187 forks source link

Question about using proxy in WebCrawler #52

Closed xwt1 closed 1 month ago

xwt1 commented 2 months ago

Hi, I'm currently using WebCrawler to scrape websites, and I need to set a proxy. I've tried using the os.environ method to set a global proxy, Unfortunately, this method did not work as expected. Could you please provide guidance on how to correctly set a proxy for WebCrawler? Any help will be appreciated! Thank you!

unclecode commented 1 month ago

@xwt1 Thx for using the library, let me work on that and will back to you

xwt1 commented 1 month ago

Thanks for reply! Your amazing work truly help me a lot!