Closed reverieinc123 closed 3 years ago
zimmerle
commented
3 years ago Hi @reverieinc123,
The ModSecurity is just the engine. The inspections are held by the rules that you have loaded. What you have loaded is a custom rule or some public rule set?
reverieinc123
commented
3 years ago Actually we have used SEMRush tool to audit our website www.reverieinc.com but its shows an error called blocking SEMRush bot crawler. We have contacted the SEMRush team as they mentioned The software can block our bots from crawling a website & thus prevent users from fully benefitting from our platform's functionality.
You may contact Mod Security in order to whitelist our bot.
I would like to understand why its happening
zimmerle
commented
3 years ago @reverieinc123,
ModSecurity process a ruleset against a request in order to asses if a request looks like something malicious or not. The one who classify a request as malicious is the ruleset. Somewhere in your ModSecurity configuration file you have a rule that is blocking the requests from this crowler.
Usually, when such thing happens it is the case to put the bot in some sort of allow list. Impossible to tell what to do without have the details on your ruleset. Can you provide the error message? The information on the rule set?
reverieinc123
commented
3 years ago Dear Team,
FYI, attached log report.
Regards, Sudhakar
On Fri, Dec 11, 2020 at 10:42 PM Felipe Zimmerle notifications@github.com wrote:
@reverieinc123 https://github.com/reverieinc123,
ModSecurity process a ruleset against a request in order to asses if a request looks like something malicious or not. The one who classify a request as malicious is the ruleset. Somewhere in your ModSecurity configuration file you have a rule that is blocking the requests from this crowler.
Usually, when such thing happens it is the case to put the bot in some sort of allow list. Impossible to tell what to do without have the details on your ruleset. Can you provide the error message? The information on the rule set?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SpiderLabs/ModSecurity/issues/2475#issuecomment-743316991, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASCJ2THP4JGTZAIHW5GBWTTSUJHILANCNFSM4UVHOFFA .
-- The information contained in this e-mail message and/or attachments are confidential or privileged information of Reverie Language Technologies Pvt. Ltd. Unauthorized dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments.
IP addresses of Semrush Bot ( Site Audit ): 46.229.173.66, 46.229.173.67, 46.229.173.68
REQUEST: curl -i -sS -L --proto-redir -all,http,https --max-time 5 -A 'Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)' http://reverieinc.com/
RESPONSE: HTTP/1.1 406 Not Acceptable Date: Fri, 11 Dec 2020 05:57:58 GMT Server: Apache Content-Length: 226 Content-Type: text/html; charset=iso-8859-1
An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.
Describe the bug We can't able to audit website using SEMRush tool in order to do that we need to whitelist below mentioned IPs.
Domain: www.reverieinc.com
Logs and dumps
User-agent: Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)
IP addresses: 46.229.173.66, 46.229.173.67, 46.229.173.68