Wrong regex

[emanruse]

This:

RewriteCond %{HTTP_USER_AGENT} ^360Spider.* [NC,OR]

will not match: "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider"

This:

RewriteCond %{HTTP_USER_AGENT} ^Dotbot.* [NC,OR]

will not match:

"Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"

[mitchellkrogza]

Thanks @emanruse do you have a better regex for this? I do not use Apache anymore myself but still support it and would like to keep this running for others.

[emanruse]

RewriteCond %{HTTP_USER_AGENT} ^(.*)(\bDotbot\b).* [NC,OR]

[emanruse]

Perhaps even this would work (as the leading and trailing .* really only burden the regex):

RewriteCond %{HTTP_USER_AGENT} \bDotbot\b [NC,OR]

[mitchellkrogza]

Sorry for the delayed response on this been so busy with other projects. Will test this out today and tag you so you can test changes.

[davcpas1234]

@mitchellkrogza I can confirm from testing that @emanruse \b<STRING>\b regex works

[mitchellkrogza]

Thanks @davcpas1234 I will push out a modified a fix now now so you and @emanruse can test.

[mitchellkrogza]

@davcpas1234 @emanruse please can you test latest updates

https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/_htaccess_versions/htaccess-mod_rewrite.txt

https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/_htaccess_versions/htaccess-mod_setenvif.txt

Also

should we not improve the regex for %{HTTP_REFERER} too or is that working ok?

ie. RewriteCond %{HTTP_REFERER} ^http(s)?://(www.)?.*prizestohandle\.club.*$ [NC,OR] to RewriteCond %{HTTP_REFERER} \bprizestohandle\.club\b [NC,OR]

[davcpas1234]

@mitchellkrogza - Happy to confirm solution from a bot perspective:

In regards to %{HTTP_REFERER} suggestion, the current format works well from my perspective as it eliminates sub-domains and sub-directories.

[mitchellkrogza]

Thanks so much David that's great we will stick with the current format. Appreciate you testing and helping to improve things for the project.