crawler-user-agents extende list

[fepe89]

Hi Thomas, thank you for your work. I have an additional list from monperrus – crawler user agent https://github.com/monperrus/crawler-user-agents I´,ve put it in the right form to upload it. The list is very accurate so i wonder, if u want to fetch it in automatically? If not, i added the file, so maybe you want to add it to the repo so that other users can extend the list very easy if they want to? I hope that helps. crawler-user-agents-extend-list.txt

[Thomas--F]

Hi fepe89, first of all: thank you for the extendet list and the effords you made.

But I have to clean some of the entrys:

• Because "googlebot" is in the default-list, all of the bots that contain "googlebot" are useless duplicates.
• If I search for "AdsBot-Google", I will also find "AdsBot-Google-Mobile", so I don't need the second entry
• The search runs by string compare, so RegEx like "([^-]|$)" doesn't work.
• The compare is not case sensitiv, so it is not necessary to search "Wget" and "wget"
• The import fails, if there are aditional vertical lines in the keywords.

I tried to remove all of these issues an added the rest to the botlist.

[Thomas--F]

botlist.txt is now fixed