search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Convert user-agent to lowercase on _get_matching_rule_set #41

Closed VMRuiz closed 1 year ago

VMRuiz commented 1 year ago

Conver user-agent to lowercase before searching in _matched_rule_set were all user-agents keys are stored as lowercases

VMRuiz commented 1 year ago

Not a bug, I wasn't understanding the logic correctly