search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Test for malformed disallow directives #3

Closed maramsumanth closed 4 years ago

maramsumanth commented 4 years ago

There are other variations of useragent, sitemap, crawl delay and request rate (as mentioned in protego.py) which can be tested.