Rules not handled correctly if dont't find User Agent

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

http://robots.jeka.by

MIT License

44 stars 17 forks source link

Rules not handled correctly if dont't find User Agent #34

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

If specify User Agent when call isUrlAllow() and the User Agent is not find in the content of the robots.txt file, isUrlAllow() not handled correctly if there is User-agent: * in content of robots.txt.

Robots.txt file

User-agent: *
Disallow: /fr/

$robotsTxtValidator->isUrlAllow('http://site.com/fr/page2', 'MyUserAgent') => return True