Fix disallow not working when no path is provided

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.

BSD 3-Clause "New" or "Revised" License

54 stars 28 forks source link

Closed Laerte closed 2 years ago

Laerte commented 2 years ago

This close #17

I try a simple approach, when we don't have any path we return the default /

Gallaecio commented 2 years ago

Nice work, thanks!