search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Enforce path only in Alllow/Disallow #8

Closed whalebot-helmsman closed 4 years ago

whalebot-helmsman commented 4 years ago

After discussion @Gallaecio and I decided not to implement full URLs handling. To fix https://github.com/scrapy/protego/issues/4 and https://github.com/scrapy/scrapy/issues/4145 we enforce / at the start of Disallow/Allow patterns.

Also added ability to pass arguments from tox to pytest.