Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Adds support for decompressing a brotli-compressed server response if an "Accept-Encoding" header was passed with the "br" or "gzip, deflate, br" parameter and the server supports this type of compression.
Httpx provides the necessary functionality, but requires additional libraries to be installed to work correctly.
Description
Adds support for decompressing a brotli-compressed server response if an "Accept-Encoding" header was passed with the "br" or "gzip, deflate, br" parameter and the server supports this type of compression.
Httpx provides the necessary functionality, but requires additional libraries to be installed to work correctly.