search

BedrockStreaming / roboxt

:no_entry: DEPRECATED - simple parser for robots.txt
19 stars 7 forks source link

User-agent names case-insensitive match #8

Open ranvis opened 8 years ago

ranvis commented 8 years ago

According to the draft http://www.robotstxt.org/norobots-rfc.txt User-agent names are case-insensitive. Currently User-agent of 'Crawler' cannot read the record like:

User-agent: crawleR
Disallow: /no

Also if no matching record is found (i.e. no '*' record), the following error is occurred when you call isUrlAllowedByUserAgent(). Error: Call to a member function isUrlAllowed() on null in /path/to/vendor/m6web/roboxt/src/Roboxt/File.php:90

ranvis commented 8 years ago

Forget to add that when the '' record is missing the following notice is also triggered: PHP Notice: Undefined index: \ in /path/to/vendor/m6web/roboxt/src/Roboxt/File.php on line 73