Open johnnysprinkles opened 6 years ago
I was hoping to use this library to detect search spiders/robots... and I see a family of "Spider" listed in the regexes. But when I parse say Googlebot's user-agent I get the name of the bot itself for family.
With any of these:
useragent.parse('Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'); useragent.parse('Googlebot/2.1 (+http://www.googlebot.com/bot.html)'); useragent.parse('Googlebot/2.1 (+http://www.google.com/bot.html)');
I get family: 'Googlebot'
Any way to detect any and all spiders in a general way?
I was hoping to use this library to detect search spiders/robots... and I see a family of "Spider" listed in the regexes. But when I parse say Googlebot's user-agent I get the name of the bot itself for family.
With any of these:
useragent.parse('Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'); useragent.parse('Googlebot/2.1 (+http://www.googlebot.com/bot.html)'); useragent.parse('Googlebot/2.1 (+http://www.google.com/bot.html)');
I get family: 'Googlebot'
Any way to detect any and all spiders in a general way?