search

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications
http://robots.jeka.by
MIT License
44 stars 17 forks source link

Remove comments in content #25

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

For optimize parsing, I remove comments in content. I do this with regexp like this :

$content = file_get_contents('http://example.com/robots.txt');
$content = preg_replace('/^#.*\\n/m','', $content);
$parser = new RobotsTxtParser($content);
bopoda commented 7 years ago

@LeMoussel comments was removed with preg_replace when RobotsTxtParser has been refactored in https://github.com/bopoda/robots-txt-parser/pull/39. $row = preg_replace('/#.*/', '', $row);

Thanks!