search

jenssegers / agent

👮 A PHP desktop/mobile user agent parser with support for Laravel, based on Mobiledetect
https://jenssegers.com
MIT License
4.54k stars 476 forks source link

GrapeShot Crawler not recognized as bot #57

Open sstringer opened 8 years ago

sstringer commented 8 years ago

I am using v.2.3 in Laravel 5.2.

The GrapeShot crawler using the following user agent is not recognized as a bot:

Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; http://www.grapeshot.co.uk/crawler.php)

It is incorrectly being returned as a desktop browser, and the device, browser, and other tests are all blank.

Screenshot of debug test results here: https://www.evernote.com/l/AAMvzzMfUW1FOpu9pMk6n29vJqXYDWE0g3s

sstringer commented 8 years ago

Following up, there are tons of similar issues where bots aren't being properly detected. Just in my own logging in the past few days, these include (but are hardly limited to):

JayBizzle commented 7 years ago

The isBot() method is by no means exhaustive in MobileDetect. I mainly catches the 'big boys'...Google, Bing etc.

If you need a more specific bot detector, then you could use MobileDetect in combination with CrawlerDetect.

jenssegers commented 7 years ago

We could integrate that library into this one to provide better bot detection. What do you think @JayBizzle ?

JayBizzle commented 7 years ago

Yes, I would be open to that 👍