List of possible bots to check

sgiehl commented 4 years ago

List of useragents from https://github.com/matomo-org/matomo-log-analytics/pull/239 that are currently not detected as bot (or library):

[ ] Hatena-Favicon2 (http://www.hatena.ne.jp/faq/)
[ ] Mozilla/5.0 (X11; Linux x86_64; rv:20.0; Favicon; +https://github.com/ArthurHoaro/favicon) Gecko/20100101 Firefox/32.0
[ ] com.ddeville.llwebkit.favicon/158 CFNetwork/902.1 Darwin/17.7.0 (x86_64)
[ ] com.ddeville.llwebkit.favicon/158 CFNetwork/974.1 Darwin/18.0.0 (x86_64)
[ ] Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.170 Safari/537.36 Thumbshots/10.0.0.0
[ ] Thumbor/6.4.2
[ ] rethumb/v1 (http://rethumb.com)
[ ] FetchStream
[ ] Hatena::Fetcher/0.01 (master) Furl/3.13
[ ] Hatena::Fetcher/0.01 (master) Furl/3.06
[ ] safarifetcherd/604.1 CFNetwork/974.2.1 Darwin/18.0.0
[ ] safarifetcherd/604.1 CFNetwork/975.0.3 Darwin/18.2.0
[ ] safarifetcherd/604.1 CFNetwork/901.1 Darwin/17.6.0
[ ] safarifetcherd/604.1 CFNetwork/976 Darwin/18.2.0
[ ] safarifetcherd/604.1 CFNetwork/958.1 Darwin/18.0.0
[ ] HatenaBookmark/0.03 (compatible; entryimage-fetcher)
[ ] Mozilla/5.0 (compatible; ImageFetcher/7.0; +http://images.weserv.nl/)
[ ] Mozilla/5.0 (compatible; ImageFetcher/8.0; +http://images.weserv.nl/)
[ ] Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; BacklinkHttpStatus)
[ ] Hatena Star UserAgent/2
[ ] Hatena-Favicon2 (http://www.hatena.ne.jp/faq/)
[ ] Hatena::Fetcher/0.01 (master) Furl/3.13
[ ] Hatena::Fetcher/0.01 (master) Furl/3.06
[ ] HatenaBookmark/0.03 (Hatena::Bookmark; master;) Furl/3.13
[ ] HatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)
[ ] Hatena::Scissors/0.01
[ ] HatenaBookmark/4.0 (Hatena::Bookmark; Scissors)
[ ] HatenaBookmark/0.03 (compatible; entryimage-fetcher)
[ ] Elytra/0.10.0 (Macintosh; Ubuntu/14.06) GCDHTTPRequest
[x] Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
[x] request.js
[ ] Zend_Http_Client
[ ] EventMachine HttpClient
[ ] HTTPClient/1.0 (2.6.0.1, ruby 2.0.0 (2015-12-16))
[ ] HTTPClient/1.0 (2.8.3, ruby 2.2.1 (2015-02-26))
[x] Jakarta Commons-HttpClient/3.1
[ ] Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
[ ] ~AppEngine-Google; (+http://code.google.com/appengine; appid: s~readability-api-hrd)~
[ ] ~AppEngine-Google; (+http://code.google.com/appengine; appid: e~finscience-1253)~
[ ] ~AppEngine-Google; (+http://code.google.com/appengine; appid: s~cdn-dinoia)~
[ ] ~GAE AppEngine-Google; (+http://code.google.com/appengine; appid: s~ga-mozilla-org-prod-001)~
[ ] ~Mozilla/5.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~mendoapp1)~
[ ] ~Mozilla/5.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~vodio-app)~
[ ] ~Mozilla/5.0 AppEngine-Google; (+http://code.google.com/appengine; appid: e~finscience-1253)~
[ ] ~Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~xiaohe18675)~
[ ] ~Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~thgntk2)~
[ ] ~Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~kindle-11)~
[ ] ~Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedkin)~
[ ] ~Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~ebook-rexdf)~
[ ] ~Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)~
[ ] ~Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~simple-rss-proxy)~
[ ] ~Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729) AppEngine-Google; (+http://code.google.com/appengine; appid: s~theajaxpost)~
[ ] ~Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~twilinks123)~
[ ] ~Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: b~thatkindleear)~
[x] Evergreen (macOS; RSS Reader; https://ranchero.com/evergreen/)
[x] Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/602.3.12 (KHTML, like Gecko) NetNewsWire/3.3.2
[x] Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/534.57.7 (KHTML, like Gecko) NetNewsWire/3.3.2
[x] NetNewsWire (macOS; RSS Reader; https://ranchero.com/netnewswire/)
[x] NetNewsWire/3.3.2 (Mac OS X; http://netnewswireapp.com/mac/; gzip-happy)
[x] NetNewsWire/4.1.0 (Mac OS X; http://netnewswireapp.com/mac/; gzip-happy)
[x] Evergreen (macOS; RSS Reader; https://ranchero.com/evergreen/)
[ ] Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 AppEngine-Google; ( http://code.google.com/appengine; appid: s~simple-rss-proxy)
[x] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QuiteRss/0.18.12 Safari/538.1
[x] NetNewsWire (macOS; RSS Reader; https://ranchero.com/netnewswire/)
[ ] rss/10 CFNetwork/976 Darwin/18.2.0
[ ] rss/5 CFNetwork/897.15 Darwin/17.5.0
[ ] rss/5 CFNetwork/902.2 Darwin/17.7.0
[ ] rss/5 CFNetwork/974.2.1 Darwin/17.7.0
[ ] rss/8 CFNetwork/974.2.1 Darwin/18.0.0
[ ] rss/8 CFNetwork/975.0.3 Darwin/17.7.0
[ ] ruby:net.bhaak.rss2html:1.0
[ ] Visor News Reader RSS/1.0 (admin@visorco.com)
[ ] Winds: Open Source RSS & Podcast app: https://getstream.io/winds/
[ ] RSSOwl/2.2.1.201312301314 (Windows; U; en)
[ ] Ruby
[x] Ruby, Twurly v1.1 (https://twurly.org)
[ ] ruby:net.bhaak.rss2html:1.0
[ ] HTTPClient/1.0 (2.6.0.1, ruby 2.0.0 (2015-12-16))
[ ] HTTPClient/1.0 (2.8.3, ruby 2.2.1 (2015-02-26))
[x] Mozilla/5.0 (Mixnode) AppleWebKit/537.36 (KHTML, like Gecko)
[ ] node-superagent/3.8.2
[ ] node-superagent/2.3.0
[ ] node-superagent/0.18.2
[ ] node.io
[ ] node.js
[ ] Node.js (linux; U; rv:v6.9.1) AppleWebKit/537.36 (KHTML, like Gecko)
[x] http.rb/4.0.0
[x] http.rb/3.3.0
[x] http.rb/3.0.0
[x] Mozilla/5.0 (compatible; adscanner/)
[x] Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta; +https://www.htmlyse.com/)
[ ] httpscheck (unknown version) CFNetwork/897.15 Darwin/17.5.0 (x86_64)
[ ] httpscheck (unknown version) CFNetwork/893.13.1 Darwin/17.4.0 (x86_64)
[ ] httpscheck (unknown version) CFNetwork/976 Darwin/18.2.0 (x86_64)
[ ] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 (compatible; linkCheckV3.0)
[ ] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) wkhtmltoimage Safari/538.1
[ ] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
[ ] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
[ ] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) wkhtmltoimage Version/7.0 Safari/538.1
[ ] Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
[ ] Houdini%20Sweeper/3000 CFNetwork/893.13.1 Darwin/17.4.0 (x86_64)
[ ] Houdini%20Sweeper/3000 CFNetwork/897.15 Darwin/17.5.0 (x86_64)
[ ] GoScraper
[ ] MetadataScraper
[ ] Embed PHP Library
[ ] Embed PHP library
[ ] SnowHaze Search/1.0 support@snowhaze.com
[ ] WordupInfoSearch/1.0
[ ] WordupinfoSearch/1.0
[x] Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
[x] Iframely/0.8.5 (+http://iframely.com/;)
[x] Iframely/1.2.2 (+http://iframely.com/;)
[x] Iframely/1.2.5 (+http://iframely.com/;)
[x] Iframely/1.2.7 (+http://iframely.com/;)
[x] Iframely/1.0.4 (+http://iframely.com/;)
[x] Iframely/1.2.7 (+https://iframely.com/;)
[ ] G-i-g-a-b-o-t
[ ] B-l-i-t-z-B-O-T/6.3 (Ubuntu 7.0; zh_TW;)
[ ] B-l-i-t-z-B-O-T/4.3 (AmigaOS 2.3; fr_LU;)
[ ] B-l-i-t-z-B-O-T/1.9 (Windows Vista 5.7; fr_CA;)
[ ] inbound.li parser
[x] Mozilla/5.0 (compatible; WebDataStats/1.0 ; https://webdatastats.com/policy.html)
[x] NetTrack Anonymous Web Statistics https://nettrack.info/support.php
[x] CakePHP
[ ] Bots that Matomo catches in the server (but not the log analyzer)
[ ] Mozilla/5.0 (compatible; GoogleDocs; documents; +http://docs.google.com)
[ ] Mozilla/5.0 (compatible; GoogleDocs; +http://docs.google.com; +Google-Document-Conversion)
[ ] Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; +http://docs.google.com)

Note: have not yet checked which of this useragents are actually bots or maybe valid mobile apps or other stuff....

sanchezzzhak commented 4 years ago

[x] facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) (service bot)
[x] Mozilla/5.0 (compatible; Plukkie/1.7; http://www.botje.com/plukkie.htm) (search bot)
[x] Grammarly/1.0 (http://www.grammarly.com) (service bot)

arcanedev-maroc commented 4 years ago

There are some bots missing compared to JayBizzle/Crawler-Detect

sanchezzzhak commented 4 years ago

The Mighty Website Monitoring Service

[x] montastic-monitor http://www.montastic.com (noticed activity 2020-06)

sanchezzzhak commented 3 years ago

[x] Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)
[x] Mozilla/5.0 (compatible; DataXu/1.0; +http://dataxu.com)

chris-y commented 2 years ago

Jakarta Commons-HttpClient is in that list, but I have one with User-Agent : Jakarta Commons-HttpClient/3.0.1 which isn't being recognised on devicedetector.net.

sanchezzzhak commented 2 years ago

devicedetector.net Not updated to the latest version We will fix this as soon as there is time

Try an alternative demo URL https://devicedetector.lw1.at/Jakarta%20Commons-HttpClient%2F3.0.1

andre-kornetzky commented 1 year ago

Iframely/1.3.1 (+https://iframely.com/docs/about) Atlassian

sanchezzzhak commented 1 year ago

@andre-kornetzky, this bot was added on 05/22/2022. update your version.

user_agent: Iframely/1.3.1 (+https://iframely.com/docs/about) Atlassian
bot:
  name: Iframely
  category: Crawler
  url: https://iframely.com/
  producer:
    name: Itteco Software, Corp.
    url: https://iframely.com/

matomo-org / device-detector

List of possible bots to check #6315