Include spiders/bots - Githubissues

shon / httpagentparser

Python HTTP Agent Parser

http://pypi.python.org/pypi/httpagentparser/

MIT License

222 stars 55 forks source link

Closed reiven closed 10 years ago

reiven commented 10 years ago

It will be nice to add a new type of "browser" for bot/spiders like this:

"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

shon commented 10 years ago

Yes it is desired. I wish somebody contribute with adding tests for bots.

Offbeatmammal commented 10 years ago

would love to see a split out for Googlebox as well - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html

pepijndevos commented 10 years ago

Added in #52

shon commented 10 years ago

We detect bots now. Thanks to @pepijndevos 's contributions. New version uploaded.

pepijndevos commented 10 years ago

It seems you did not merge windows phone and some tests?

shon commented 10 years ago

Latest release has it. You might want to take look. Open a new issue/push req in case there still something missing.

pepijndevos commented 10 years ago

Looks good. Thanks.