Suppress space in User_Agent

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

http://robots.jeka.by

MIT License

44 stars 17 forks source link

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

If user-agent has a spacing in its name, robots-txt-parser remove spaces. Is it normal ?

User-agent:Googlebot Disallow:/ User-agent:Cocon.Se Crawler Disallow:/

Got 'cocon.secrawler' in array result of getRules()

[cocon.secrawler] => Array
    (
        [disallow] => Array
            (
                [0] => /
            )
    )

bopoda commented 7 years ago

@LeMoussel hey, Thanks for feedback! Agree.

Now will return user-agent in lower case without removed whitespaces "as is" - PR.