WhichBrowser / Parser-PHP

Browser sniffing gone too far — A useragent parser library for PHP
http://whichbrowser.net
MIT License
1.8k stars 237 forks source link

Add `PetalBot` to bots list #544

Closed summercms closed 3 years ago

summercms commented 4 years ago

See github issue: https://github.com/WhichBrowser/Parser-PHP/issues/540

Some notes on this bot.

This bot says it's from aspiegel and from day one we have seen it coming from HUAWEI CLOUD servers! After the USA started banning Huawei they admitted that this bot is from them and it's going to be there search engine, see here: https://consumer.huawei.com/en/mobileservices/search/

Screenshot as they will probably take down their webpage:

image

Since the USA started trying to ban Huawei and TikTok etc. this bot has been hitting our test servers over 1000 times a day!

We label this bot as bad and their user agent keeps changing as time goes on. Sometimes they add fake user agents to hide, yet the same ip address then hits your server again in less than a second and then displays a user agent.

The regex code in this pr we have added code to allow a bot version number. In the future PetalBot may be like this: PetalBot/1.0 therefore we decided to add a version number now.

Note this bot is always changing and is crawling the internet to gather more information than it needs to do it's search engine job! Clearly it is data gathering for the Chinese government. The search engine aims to crawl apps only! Yet it keeps coming back to our test servers hitting them thousands of times a day!


User agents hitting our test servers:

UA:

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)

(compatible;PetalBot;+https://aspiegel.com/petalbot)
coveralls commented 4 years ago

Coverage Status

Coverage increased (+0.02%) to 99.97% when pulling fc122e904b68d90a6b4f52b37eebfb5df22eeb38 on ayumi-cloud:PetalBot into 880b9fa797401d14b28956442944c3daa70240ff on WhichBrowser:master.

NielsLeenheer commented 3 years ago

Fixed in 6628eac7dcaf3fd81c1b67fe0c6fac0089c0cd36