search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

fixed leading double slash issue #45

Open s-ferri-fortop opened 11 months ago

s-ferri-fortop commented 11 months ago

https://github.com/scrapy/protego/issues/44