Scrapy, a fast high-level web crawling & scraping framework for Python.
BSD 3-Clause "New" or "Revised" License
51.16k
stars
10.35k
forks
source link
feat: add disallowed_domains option to OffsiteMiddleware #5922
Open
felipecustodio opened 1 year ago
Resolves #2376.
There is most likely a cleaner way to write the checks for the existence of the lists/arguments!
TODO:
tests/test_spidermiddleware_offsite.py