search

aws-solutions / aws-waf-security-automations

This solution automatically deploys a single web access control list (web ACL) with a set of AWS WAF rules designed to filter common web-based attacks.
https://aws.amazon.com/solutions/aws-waf-security-automations
Apache License 2.0
845 stars 361 forks source link

Probe and Scan parser should white list a domain list on reverse resolution #86

Closed dhardy92 closed 1 year ago

dhardy92 commented 5 years ago

Some search engine are legitimate to scan ours (Bing Google Qwant ...) and they might be blocked by probe and scan log parser if there is too much Error code (4xx) per minute.

It could be usefull for SEO to prevent those bot from being block by doing some reverse resolution of IP and white list corresponding IP ranges ?

Ex: 157.55.39.224 is making too much 404

$ host 157.55.39.224
224.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-224.search.msn.com.

In config I whitelisted "search.msn.com."

NetRange        157.54.0.0 - 157.60.255.255
CIDR            157.56.0.0/14, 157.60.0.0/16, 157.54.0.0/15
NetName         MSFT-GFS
NetHandle       NET-157-54-0-0-1
Parent          NET157 (NET-157-0-0-0-0)
NetType         Direct Assignment
OriginAS        AS8075
Organization    Microsoft Corporation (MSFT)
RegDate         1994-04-27
Updated         2013-08-20
Ref             https://rdap.arin.net/registry/ip/157.54.0.0
[...]

I would like to automatically add 157.56.0.0/14, 157.60.0.0/16, 157.54.0.0/15 in whitelist IP range.

Some references : https://www.bing.com/webmaster/help/verifying-that-bingbot-is-bingbot-3905dc26 https://support.google.com/webmasters/answer/80553?hl=en

hvital commented 5 years ago

Thanks for sharing your feedback @dhardy92 .

This was included to the solution roadmap and we'll post an update to this thread once we have more information.

aijunpeng commented 1 year ago

Closing the ticket due to low customer demand. The out-of-box solution provides a starting point for implementing AWS WAF rules. We recommend you to add custom rule or customize source code to meet your company’s needs.