search

ess / citadel

Citadel is a replacement for dos-deflate (ddos.sh) implemented in Perl.
http://brokenmoon.net/citadel
GNU General Public License v3.0
12 stars 4 forks source link

Doesn't work with reverse proxy setup #7

Closed petakcm8 closed 8 years ago

petakcm8 commented 9 years ago

Hey guys,

So I've been using citadel on my server for some tests and all was working perfectly fine using apache and X-Forwarded-For , the script was getting all the right ips this way .

However, once I moved to nginx with a reverse proxy setup, it s only showing the reverse proxy ip although the X forwarded is enabled in Nginx conf and I can see the ips in my nginx logs.

Any way around this ?

Thanks

ess commented 9 years ago

@petakcm8

Hey there. This script gets the list of possibly offending ips via a call to the netstat CLI utility and does not consider either Apache nor Nginx, so the nature of the reverse proxy really shouldn't matter.

Is citadel failing to block any IPs, or is it doing something like blocking the RP endpoint?

petakcm8 commented 9 years ago

Hello ess,

Thanks for super fast reply !

The problem is that it's only showing the reverse proxy ip, which I guess it's normal since all requests go through it.

In order to get the real ips myself I have to do so by using X-Forward with my nginx setup, so I can see the real ips in the access log.

I guess the only solution would be to tweak the script to check against logs perhaps ?

ess commented 9 years ago

Could you tell me a bit more about your infrastructure/architecture? For example, are nginx and the application to which you're RPing running on the same server? If they run on separate servers, are the application endpoints accessible to the public, or are they firewalled off?

Given their configurable nature, analyzing nginx/apache logs for IPs to block may actually be outside the scope of this project, but the infrastructure information may help us to point you in the right direction to use either this project or one better suited to your needs.

petakcm8 commented 9 years ago

Here is how this is setup :

My web server DNS is pointing to an external server acting as the Reverse Proxy, which tunnels all the traffic to my backend .

The Reverse Proxy is set up with X-Forward HTTP Header support so I am able to see the ips in my nginx access logs, which otherwise don't show in my server connections which shows the reverse proxy ip for all requests.

Am not sure what you mean by application endpoints but they are not accessible to the public I guess.

ess commented 9 years ago

Thanks for the information. The back-end is what I meant as "application endpoints," it would appear.

Are you running citadel on the web server, on the back-end, or both?

petakcm8 commented 9 years ago

Just on my web server / backend.

The reverse proxy has its own script for blocking ips, but sometimes some of the traffic goes through and since this script is written in perl and I know they are using bash, I believe it's faster to mitigate the ips when/if they reach backend.

Just came across this :

http://blog.mattbrock.co.uk/get-a-continuously-updating-display-of-client-ip-addresses-on-a-web-server-using-x-forwarded-for/

Could the script be modified to parse that instead of netstat , or do you have any other suggestion ?

ess commented 9 years ago

I'll let @sgsullivan (who is basically the maintainer these days) weigh in on that possibility.

That said, I'm still not sure that I understand your infrastructure layout, but I would like to share with you what I'd personally do in this instance. It appears that the problem isn't actually that you're getting a ton of connections to your web/back-end servers (which is what citadel mitigates), but you're getting unwanted traffic to your actual application/site/what have you. I'd firewall the back-end servers off (whitelisting the external server) and consider one of the following approaches to get around bad traffic:

petakcm8 commented 9 years ago

Thank you very much, I believe this is doable since it basically just parses netstat for the total amount of ips. I just actually tried ngrep and it's getting all the x forwarded ips so it should be only a matter of changinx the regex in the script and should work.

I actually do get plenty of connections that will still go through because although the reverse proxy blocks a part of it, there are many javascript advanced floods nowadays that I keep getting with and they can easily bypass cookie and javascript challenges such as the cloudflare one.

I'm already using cloudflare mostly for cdn features, traffic pattern detection , browser integrity check and captchas, but the under attack m ode for instance is pretty easy to bypass.

I also have limits in place in my nginx setup for requests to my main location and php files, but I will have a look at Rails since I am not familiarized with this and have never used it before.

Once again, thank you so much for your time and all your valuable feedback.

Best luck for all your projects !

ess commented 9 years ago

@petakcm8 If you should modify citadel itself to use ngrep, I'd totally love to see a pull request. Off the top of my head, if this line were changed so it either used ngrep or used the contents of a configuration variable for the external utility that gets IPs, I'd probably accept the request outright ;)

petakcm8 commented 9 years ago

Well I just did so but now it's giving me the following error after editing code and adding :

for my $line ( split /^/, ngrep -il -d eth0 -W byline "x-forwarded-for" "port 80" | grep -i x-forwarded-for ) { if ( $line =~ /^X-Forwarded-For:\s+(\d+.\d+.\d+.\d+)/i ) { my $ip = $1;

Error : [ Thu Sep 10 23:00:01 2015 ] INFO: END_RUN; Shutting down. [ Thu Sep 10 23:01:01 2015 ] CRIT: Refusing to run, non stale lock file [/var/lock/citadel.lock] present.

I delete the lock file but it keeps coming back .

sgsullivan commented 9 years ago

We don't want to run system tools like grep IMO unless its actually required (which grep never is, you can use perl for this..).

Anyways this is of course feasible. My thoughts are to do this in pure perl, not calling extra system commands like ngrep or grep for that matter. I believe this should be possible with some modules on cpan (will look). I would also want to implement this by adding a option to citadel.conf, which would tell citadel whether to look at X-forwarded-for or not.

I can't promise i will be able to make this change this week, but I should have time to do so this month anyway.

petakcm8 commented 9 years ago

sgsullivan yes you are right ofcourse, but the output of ngrep without grep is too complex and messy , I don't know much about regexes but am sure you can pull something out with perl :)

Adding X-forwarded as an option in citadel.conf would be a really nice addon to this.

By the way the script works wonderfully well for me, have tested my backend against massive requests (500k+ R/S from 150 Ips) and all ips got mitigated in a few seconds when running script with a 10 second interval.

For sure much more efficient than then bash version .

Looking forward to your update

Thanks !

sgsullivan commented 9 years ago

Thanks, i will update this issue once ive had time to code and push the feature addition. On Sep 10, 2015 5:59 PM, "petakcm8" notifications@github.com wrote:

sgsullivan yes you are right ofcourse, but the output of ngrep without grep is too complex and messy , I don't know much about regexes but am sure you can pull something out with perl :)

Adding X-forwarded as an option in citadel.conf would be a really nice addon to this.

By the way the script works wonderfully well for me, have tested my backend against massive requests (500k+ R/S from 150 Ips) and all ips got mitigated in a few seconds when running script with a 10 second interval.

For sure much more efficient than then bash version .

Looking forward to your update

Thanks !

— Reply to this email directly or view it on GitHub.

petakcm8 commented 9 years ago

Just an update on this .

Seems like ngrep acts the same way as tcpdump and shows the incoming packets in real time, not a list of all open connections.

Although it is possible to make it show a certain number of packets as limit, it doesn't output an instant list of connections opened but rather shows all incoming packets.

For testing purposes, I tried running the following script :

while true ; do /usr/local/bin/citadel ; timeout 12 ngrep -il -d eth0 -W byline "x-forwarded-for" "port 80" | grep -i x-forwarded-for >> /etc/iplogs; sleep 10; done timeout 20 echo > /etc/iplogs

Citadel seems to work this way and I adjusted the netstat line to parse the log file getting ngrep stdout output during the last 10 seconds.

This approach has ofcourse its limitations, but I can't imagine other way to achieve this without parsing either nginx logs or ngrep dump over a period of time .

sgsullivan commented 8 years ago

Just to update, I haven't been able to get as much time as i've wanted to look at this. However looking at this again, I had to take the time and add tests to this project, as it was missing. That is now complete. In creating the tests, I had to make some very small changes to the main program (mainly regarding how its packaged so it could be more easily tested). The changes should be completely transparent to a end-user however. But due to these small changes, I did tag a new version (v0.1.4).

Now that tests are in place, when I next have time I will start working on a method of inspecting packets is what I am currently thinking. Since parsing nginx or apache webserver logs is not an option (that is to insane to try and support with all the various configurations out there); I am thinking of basically adding an option to citadel.conf which will say "also inspect incoming packets on this interface, for this amount of time)" via perl NetPcap and block IPs with too many packets (over allowed_cons) by manually grabbing the HTTP header from the packet looking for X-Forwarded-for.

There is no other/better way to get that data as you also mentioned. The only portable way is to manually extract the http headers from a raw packet.

petakcm8 commented 8 years ago

That's really great news, I'm sure many people out there will find it extremely useful to have an option to ban ips behind a reverse proxy. The Layer7 DDOS Methods being used by attackers these days can bypass most protections out there, even enterprise level ones, since they manage to bypass javascript and cookie challenges, eventually reaching the backend of their targets.

Having an option to process and ban ips on the kernel level for users behing reverse proxies / cloudflare would be an extra layer of security that could really have a significant impact on successfully mitigating these attacks.

I don't believe there is currently any solution I am aware of which is able to do this for servers behind a proxy, you can always blacklist offensive ips in nginx (in my case I run a Nginx + Lua setup so the requests go through the lua module first and are handled more efficiently), but being able to block them even before they overload the server with requests would be amazing.

Do you have any idea how you could inspect packets to parse them for the X/Forwarder-for headers without using a log file ? I mean, Netpcap or other tools such as tcpdump and ngrep are getting the packets in real time, while the logic behind citadel at the moment is getting the total amount of open connections at a single moment when it checks them through netstat.

Wouldn't that imply a total change of logic by making citadel inspect all packets in real time instead of checking every x seconds ? Otherwise it would need to log packets to a file ...

sgsullivan commented 8 years ago

It would have to parse them in realtime as they come in through the network interface you give. Its either that or from a given logfile as you mentioned. I am leaning towards inspecting in realtime.

A possibility is saying in the conf file "Look for x-forward-for for this many seconds per run". And a conf file option that says "check for x-forward-for on these interface(s)". And obviously the conf file option to enable/disable the x-forward-for checks. If the user selects to have checking for x-forward-for enabled, I was thinking of having it do the current netstat check in addition to that.

For implementation I am interested in http://search.cpan.org/dist/Net-Pcap/Pcap.pm to capture the packets. Assembling would be needed though to rebuild the http headers so we can pull out X-forwarded-for.

petakcm8 commented 8 years ago

Just came across this script, not sure how useful it is but seems like a good starting point for what we want to achieve here .

http://wiki.opennicproject.org/ddosDotPl

Just thought some of the code there would be helpful

sgsullivan commented 8 years ago

Think i am just going to close this as its really outside what this simple tool ever aimed to do. If I was to solve this specific problem I would write a different tool altogether, having it tailored to just that single purpose.