$http_cf_connecting_ip

[shanemarsh28]

Hi,

This isn't a bug but a question/feature request.

I noticed you have a block list of IP addresses which is great but as we use Cloudflare to front our sites, I don't believe this part of it is working correctly. I beleive the bot blocker is only seeing the IP address of Cloudflares proxy. Ideally if the request header cf_connecting_ip exists, it needs to check against that IP address.

Is there a way to configure your bot blocker to do that?

Shane :)

[mitchellkrogza]

Hi @shanemarsh28 I've often pondered about Cloudflare and the blocker. While I did build in whitelisting of all known cloudflare ranges at the time, it's not been update nor tested either as I do not use Cloudflare. It's certainly a worthy point you raise and something worthwhile getting right.

Being a cloudflare user do you have any way to test and find a solution that can be implemented? You can just create a backup of globalblacklist.conf and then modify / test any changes and if they work send me details or send a PR for it

[shanemarsh28]

Yes I'm happy to test! I have 130+ sites running through Cloudflare. It's on my mind at the moment because we are getting a LOT of bots trying to target login pages and a few of the IP addresses that have flagged up after we parsed our access logs are already in your list! :)

I know how to get hold of the correct header so if your able to point me in the right direction in your code, I'm more than happy to have a play. I'll of course post up any code changes that work. Or, if you have any other suggestions then please fire away :)

[mitchellkrogza]

Any info you have, headers, log excerpts etc can help with looking into this. Post anything you have and I can look into it in the morning. I personally employ a number of other mechanisms to deal with bots looking for login pages but have yet to ever introduce these into the blocker as it may or may not block valid users from accessing their own stuff which we do not want

[shanemarsh28]

Well I'm seeing rather coordinated attacks on our systems and when I parsed the access logs I saw this rather interesting (Hits/IP)...

2425 89.234.36.212
2425 213.32.91.176
2425 198.1.94.160
2425 188.116.17.202
2425 185.41.162.16
2425 185.26.97.202
2425 185.13.39.28
2425 162.144.123.166
2425 148.251.207.119
2425 107.180.79.184
2425 104.238.98.12
2425 104.236.64.131
2425 103.6.245.217
2424 87.120.254.152

Always going to /wp-login.php. Our access log for today is currently running at 150Mb so it's a little tough to work with that but just to give you an idea, today we have had 67,500+ requests for the login page. As it happens we've moved it from default so our server returns a 404 but being wordpress, that still involves a database lookup and thats half the issue - it's the DB traffic. The requests come in blocks of 500-1000 requests at a time, in a distributed fashion. This bogs the server down for a couple of mins before it recovers.

So for example hears a short excerpt of a very small attempt at one of our sites Soul Hairdressing..

Line 52081: www.soulhairdressing.com 185.202.97.155 NL 141.101.77.87 - - [13/Jul/2018:03:11:01 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31" Line 52109: www.soulhairdressing.com 103.194.113.144 JP 162.158.6.77 - - [13/Jul/2018:03:11:22 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31" Line 52109: www.soulhairdressing.com 103.194.113.144 JP 162.158.6.77 - - [13/Jul/2018:03:11:22 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31" Line 52154: www.soulhairdressing.com 165.231.84.110 FR 162.158.222.23 - - [13/Jul/2018:03:11:34 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31" Line 52154: www.soulhairdressing.com 165.231.84.110 FR 162.158.222.23 - - [13/Jul/2018:03:11:34 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31" Line 52174: www.soulhairdressing.com 185.202.97.155 NL 141.101.77.87 - - [13/Jul/2018:03:11:46 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31" Line 52174: www.soulhairdressing.com 185.202.97.155 NL 141.101.77.87 - - [13/Jul/2018:03:11:46 +0100] "GET /wp-login.php?action=register HTTP/1.1" 403 135 "https://www.soulhairdressing.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17" "1.31"

So to explain after the line number its: Domain/CF-Connecting-IP/country code/CF Proxy IP -- {rest of line ...} So looking at the log, two requests from Netherlands, then two from Japan, then two from France then it goes back to Netherlands. We don't operate in any of those countries.

So, my thoughts are...

The two useful request headers Cloudflare provide are: CF-Connecting-IP - $http_cf_connecting_ip CF-IPCountry - $http_cf_ipcountry

If bot blocker can get the correct IP through via those headers, in theory, you shouldn't need to whitelist Cloudflare IP's and you can block using your IP list/setup. I would also like to rate limit (separately) /wp-login somehow. Not sure of the best way forward with that.

In your code it looks like you firstly map and then use limit_conn_zone & limit_req_zone? I'll check those functions out after the weekend - not seen them before.

[itoffshore]

The proper way to fix this is to use a vpn or fixed ip address to administer your servers & then limit access to the login page to your fixed ip address(es) in nginx.

fail2ban can also be configured to automatically block these types of offenders.

[mitchellkrogza]

👍 I have a fail2ban filter I created for this very purpose BUT PLEASE make sure to whitelist all your own IP's or you will get banned - https://github.com/mitchellkrogza/Fail2Ban.WebExploits

[itoffshore]

@mitchellkrogza - if you limit front end access to media / images / content folders in the web server configuration you can much more simply just block 403's in fail2ban

[Definition]
failregex = <HOST> -.*"(GET|POST|HEAD).*HTTP.*" 403
ignoreregex = <HOST> -.*"(GET|POST|HEAD).*HTTP.*" 200

@shanemarsh28 - running hiawatha as a reverse proxy will give you a lot more protection. It's banning ability can be used as an alternative to fail2ban

[mitchellkrogza]

@itoffshore any good guides on hiawatha for that purpose? Fail2Ban can be rather sluggish to say the least.

[itoffshore]

I started with the hiawatha notes from this site & also here from the same site.

Then read the Manual Pages & howtos

seeUrlToolkit & create one with your custom blocking rules.

Hiawatha can also detect & block exploit & garbage requests automatically.

[itoffshore]

@mitchellkrogza - use fail2ban with blocking done with ipset

banaction = iptables-ipset-proto6-allports
banaction_allports = iptables-ipset-proto6-allports

(see it's config in /etc/fail2ban/action.d)

I block 60,000 ip's with ipset with no slowdown

[shanemarsh28]

Thanks for all your great suggestions! I think fail2ban is going to be the way forward. hiawatha isn't a straight forward option due to how our "stack" is laid out. Our webservers are heavily loaded and complex.

So with regards to fail2ban, I assume you can fully customise how a ban is triggered? My only concern with it is we have a many staff and clients logged in (all in different locations and countries) - manually white listing their IP addresses would be nearly impossible and if we start having issues where our clients are getting locked/blocked out of WP Admin we'd be forced to disable it. Sadly it's not a simple case where I could just whitelist the IP address of our office and be done with it.

If anyone can point me to a tutorial or guides to help with that I'd be very grateful :)

Shane :)

[shanemarsh28]

Going back to the original part of this thread, I think I have found a solution to the Cloudflare IP problem...

https://support.cloudflare.com/hc/en-us/articles/200170706-How-do-I-restore-original-visitor-IP-with-Nginx-

[mitchellkrogza]

Thanks @shanemarsh28 I actually have that config in my Nginx but it's not been updated for some time so will update mine with their latest list. Hope all OK now your end.

[shanemarsh28]

Hi,

Great. Well I have been burning the midnight oil testing and configuring fail2ban and this morning I moved the config to one of our production servers. I tell you, it took a man with a much bigger brain than me to figure out all the complex IP4/IP6 regex but I think we've done it :) I also integrated it with Cloudflare so it blocks via the IP tables and the Cloudflare firewall so hopefully this issue is resolved for now. I did take a look at your webexploits - I'm unsure at the moment about implementing those. I think you'd have to be very careful how you set that up cos you could quite easily block genuine traffic. I'll see how we get on with the first few jails I've setup review it again :)

Thank you all so much for your help!