crawlers_detection.conf is based on information of inactive project www.useragentstring.com.
The current implementation of crawler detection is broken and has never actually worked from the beginning.
Solution
By this PR, crawler detection is replaced with regular expression matching using new information source:
monperrus/crawler-user-agents. Instead of customization through configuration file, the latest information will be available.
Problems
crawlers_detection.confis based on information of inactive project www.useragentstring.com.Solution
By this PR, crawler detection is replaced with regular expression matching using new information source: monperrus/crawler-user-agents. Instead of customization through configuration file, the latest information will be available.
This may fix #286 .