COUNTER_Robots_list.json: Add two new bots

atmire / COUNTER-Robots

Official list of user agents that are regarded as robots/spiders by COUNTER

MIT License

64 stars 29 forks source link

COUNTER_Robots_list.json: Add two new bots #43

Closed alanorth closed 3 years ago

alanorth commented 3 years ago

This adds two new bot user agents: crusty and newspaper. These two non-human user agents have made thousands of requests to my server with the following user agents:

crusty/0.12.0
newspaper/0.2.8

See the patterns file for more information about each. I have manually verified the user agents in both of these by looking at their source code.

Please let me know if you would like me to adjust the patterns to include further regex metacharacters like digits.

davidatmire commented 3 years ago

PR discussed and approved with the COUNTER working group. The following UAs were added to the exclusion list.

crusty\/\d
newspaper\/\d

The changes can be checked here.