Closed appukonrad closed 6 years ago
I'm not sure about adding new default bots. Every new entry makes the tracker slower, so only the top bots and crawlers should be in the default list. What about an additional list within this repository for every other known bot, that can be insertet with the import-function?
Make a choice @Thomas--F ^ ^ When you have choosed i will watch for add some regular bot who disturb my stats :)
Name : User Agent
Riddler (dataminer (it make full scan)) : Riddler (http://riddler.io/about) Prestashop (scan prestashop user's domain between 1am and 2am) : use some fake User Agent but IP are 91.240.109.*
I started to add new bots to a file "botlist.txt".
Feel free to add more bots to that list. Whoever wants more bots than the default can now download this list an add it with the file import at the bottom of the bot-configuration-list.
Use the file "botlist.txt" in the root-directory for aditional bots. They can be imported with the file import
Added Seznam.cz crawlers