Poor perfomance when parsing large file.

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

http://robots.jeka.by

MIT License

44 stars 17 forks source link

Poor perfomance when parsing large file. #20

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

When parsing large robots.txt file, the process can take several minutes to finish, even with 100% CPU power dedicated. On a small server, it slow down the server, and even cause it to hang or crash.

Example URL: http://www.testmateriel.com/robots.txt (20997 lines, 1 122 590 characters)

One solution is to limit the maximum number of bytes to parse (see Issue #2 ). But, perhaps, it may be possible to optimize parsing.

bopoda commented 7 years ago

I think it can`t be optimized in such look. Need try to rewrite completely :)

bopoda commented 7 years ago

http://robots.jeka.by/goldmansachs.com
http://robots.jeka.by/testmateriel.com

<1s, really fast now. earlier it took at least several minutes or more.