search

t1gor / Robots.txt-Parser-Class

Php class for robots.txt parse
MIT License
83 stars 31 forks source link

[BUG] Allowed in, while the only rule specifies Disallow: / #93

Open JanPetterMG opened 2 years ago

JanPetterMG commented 2 years ago 
$t1gor = new t1gor\RobotsTxtParser\RobotsTxtParser(<<<TXT
user-agent:*
disallow:/
TXT
);
var_dump($t1gor->isAllowed('/'));

Output: bool(true) Expected: bool(false)

Quote

Valid robots.txt lines consists of a field, a colon, and a value. Spaces are optional, but recommended to improve readability.

Source: https://developers.google.com/search/docs/advanced/robots/robots_txt#syntax