Improve the user-agent matching - Githubissues

t1gor / Robots.txt-Parser-Class

Php class for robots.txt parse

MIT License

83 stars 31 forks source link

Improve the user-agent matching #72

Closed JanPetterMG closed 8 years ago

JanPetterMG commented 8 years ago

Parsing of the user's user-agent needs an overhaul.

Verify that the format is correct
Check for invalid characters, throw an exception if needed
Better version number matching

JanPetterMG commented 8 years ago

In addition to the above:

White-space independent group determiner
Throws exceptions on seriously malformed user-agents
Tested

https://github.com/t1gor/Robots.txt-Parser-Class/commit/8f6b4e79dfd111d1295b12f874bce4d79911fe06