Closed Gallaecio closed 2 years ago
>>> from protego import Protego >>> robots_txt = "User-Agent: *\nDisallow: /\n" >>> robots_txt_parser = Protego.parse(robots_txt) >>> robots_txt_parser.can_fetch("http://example.com/", "mybot") False >>> robots_txt_parser.can_fetch("http://example.com", "mybot") True >>>
Both calls should return False, since the / path is implicit if a URL has no path.
False
/
Both calls should return
False
, since the/
path is implicit if a URL has no path.