search

unclecode / crawl4ai

🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper
Apache License 2.0
16.39k stars 1.2k forks source link

Detect different failures cases #206

Closed si2w closed 2 weeks ago

si2w commented 3 weeks ago

It would be useful as well to detect pages with:

Those errors are already handled if I'm not wrong :

unclecode commented 2 weeks ago

Thank you for using our library. I suggest you check this issue, https://github.com/unclecode/crawl4ai/issues/214, I've added a new feature that's likely the best way to handle such things by taking control of your own browser instead of the default one. You can also pass user data folder and do a lot of things. Additionally, we've added some features to help with this process. Please check and let me know if it's helpful.

unclecode commented 2 weeks ago

I closed this issue, but you can continue the conversation here or in issue #214.