Open ForestJohnson opened 10 months ago
At least this one
https://ipinfo.io/AS45102/47.76.0.0/17
has been crawling, ignoring robots.txt, and using a non-bot user agent header :angry:
FWIW in my code I put
if strings.HasPrefix("47.76", caddyLog.Request.GetRemoteIP()) { return 16 // 16: alibaba cloud }
At least this one
https://ipinfo.io/AS45102/47.76.0.0/17
has been crawling, ignoring robots.txt, and using a non-bot user agent header :angry: