Add multiple bots found in the logs

[monperrus]

Thanks, CI is failing.

[fekir]

I've removed some bots (I'll eventually readd them in a separate PR), how should I handle, for example, following error?

ValueError: Pattern 'deadlinkchecker' is a subset of 'LinkChecker'                                                                                                    

It is not a subset as the casing is different, and they are different bots, so the url field should be different.

[monperrus]

that's lots of new bots, thanks. What was your strategy to identify them?

[fekir]

I've grepped for insensitive literal "bot", "spider" and "crawler" after applying all your regexes

[monperrus]

simple and effective

[monperrus]

how should I handle, for example, following error?

the bot is already matched by the superset regexp, so we're safe.

[fekir]

how should I handle, for example, following error?

the bot is already matched by the superset regexp, so we're safe.

Ah, I see, I should execute all regexes as case-insensitive