Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Currently, runs do not fail automatically when errors occur. Instead, users must manually inspect the returned statistics object to determine if a failure has occurred and handle it themselves.
While the current setup works well for certain use cases, there are scenarios where an immediate failure is the preferable approach.
Solution
Introduce a fail_fast flag (or come up with a better name) that, when enabled, automatically terminates the run upon encountering any errors.
Description
Solution
fail_fastflag (or come up with a better name) that, when enabled, automatically terminates the run upon encountering any errors.