search

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications
http://robots.jeka.by
MIT License
44 stars 17 forks source link

Rule: noindex not handled #26

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

robots noindex meta tag is not recognized

$robotsTxtContentNoIndex = "
User-agent: *
    Noindex: /page-a.html
    Noindex: /article-*
";

$parserRobotsTxt = RobotsDotText::withContent($robotsTxtContentNoIndex);
$rules = $parserRobotsTxt->getRules();
print_r($rules);

Result :

Array ( [*] => Array() )

Should be:

    [*] => Array
        (
            [noindex] => Array
                (
                    [0] => /page-a.html
                    [1] => /article-*
                )
        )
LeMoussel commented 7 years ago

Resolve with PR #28