search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Support absolute URLs #5

Closed Gallaecio closed 4 years ago

Gallaecio commented 4 years ago

Fixes #4

whalebot-helmsman commented 4 years ago

Sorry, guys I have a very crazy week. Can we discuss it before merging?

Gallaecio commented 4 years ago

@whalebot-helmsman and I have talked in private, and he’s going to propose an alternative fix which is much simpler and more in line with what popular search engines do in these cases.