search

DPsystems / Login-Shield

Your first line of defense against Internet bots, hacks and probes. Login-Shield is a small set of bash scripts that implements an iptables/ipset blocklist of known sources of hack activity. Works great as a compliment with/without fail2ban. Statistics have shown it blocks 90+% of most system probes and attacks on login ports.
Other
36 stars 6 forks source link

Open discussion about Login-Shield #2

Open DPsystems opened 4 years ago

DPsystems commented 4 years ago

There's been some good comments about this set of scripts and what others are out there. So I thought I might as well dedicate one of these issue threads to discussing various ideas:

DPsystems commented 4 years ago

First I'm looking for more feedback on the efficacy of these blacklists and whether they interfere with any normal/legit traffic? (This will also depend on who's using it. I wholesale block as much CN and similar space as I can, so people overseas might have completely different results).

Beyond that, if the system appears to be useful and people like it, my intention would be to rewrite it in something like Perl and use a config file where people can add whitelisted IP blocks and the system will automatically remove any rule that conflicts with their whitelisting rules, and set this up in such a way that a configured system can pull new updates without having to re-do everything. But first I need to see if there's demand/interest?

Another option I want to investigate is another layer of the login-shield that affects http/https traffic. For example, if you're running a web site, what are the chances that other IP space that represents hosted servers needs to access your http/https ports? Aside from search engine crawlers and a few specific applications, I think a blacklist of hosted server platforms blocking access to http/https could dramatically cut down on attack vectors. This is something I'm also working on right now.

SomePersonSomeWhereInTheWorld commented 3 years ago 
awk: cmd. line:1: error: division by zero attempted
awk: cmd. line:1: error: division by zero attempted
awk: cmd. line:1: error: division by zero attempted
awk: cmd. line:1: error: division by zero attempted

` How do you determine the IP ranges? What prompts you to update them? Just trying to gat some insight into the workings/mechanisms.

How do you whitelist or remove a single IP address/

`For persistence after a reboot would it be as simple as running these scripts in order? Note I'm omitting 2:

create-blacklist.sh
blacklist-main-nonUS.sh
blacklist-others.sh
set-iptables.sh
DPsystems commented 3 years ago

Thanks for the feedback - I'll fix those glitches.

As far as the division by zero, that's probably because of a lack of data on file... just be patient and once you get some positive/negative entries in the log files, it should go away. (I didn't make the system so sophisticated it could avoid those errors right now. I wanted to code it mostly in shell script but a future version will probably be in Python and not have those glitches)

For persistence after a reboot would it be as simple as running these scripts in order? Note I'm omitting 2:

yes, that works. The reason I didn't include details or intentionally make it persistent is I like the idea of "turning it on" intentionally, so you can really see how much difference it makes. When people are confident in the system, they can make it persistent. Plus, I don't reboot my servers that much.

I am hoping to make it more streamlined in the future. I've been using this system in-house now for several years and seeing if there's enough interest for me to polish it more for public consumption.

How do you determine the IP ranges? What prompts you to update them? Just trying to gat some insight into the workings/mechanisms.

I monitor logs on my servers. When I encounter an attack vector I look up who is behind it. Is it coming from a hosted platform or general access space? I make certain decisions based on what I gather about where things are coming from. I also have IP blacklists that I've been developing since 1994. One of the main criteria for getting permanently on the list is how receptive/likely are the hosts to respond to abuse requests? If I feel an inquiry to the NOC can stop the attack/probe, I'm unlikely to add them to the list. If I know it's a waste of time, they get blacklisted. Plus in a general sense, I know which areas should not be accessing my server. If, for example, someone was running a server in China, Russia or Brasil, my system isn't tweaked for that, but the US and Europe, it's very well-tuned to stop a majority of unauthorized traffic.

Some of it is common sense: Is there anybody in China that needs to log into my e-mail system? No? So there's a few class A IP blocks that cover all of China? They'll end up in the list.

I've also learned from experience that certain ranges do include legit activity, so the blacklist is not by any means arbitrary. It's based on more than 20 years running mail and other services.

Stay tuned... I have another version for blocking web access that I'm testing right now. It's even more precisely targeted. I'm running multiple systems to detect nefarious activity

DPsystems commented 3 years ago

Just FYI, I was an early, principal contributor to the Spamcop system. We had a difference of opinion when they allowed people to pay them to not be in their blacklist. So I began developing my own systems.

SpareSimian commented 3 years ago

How well does this system work with LetsEncrypt's ACME system? They don't publish the IP addresses they send their queries from, so I'm worried that I might be blocking their hosts. I suppose I could hook into their renewal script to shut down Login-Shield temporarily during certificate renewal.

DPsystems commented 3 years ago

Are you talking about the automated web certificate system? Login-shield shouldn't interfere with LetsEncrypt but perhaps my web based system might. How does LetsEncrypt handshake with its servers? Using what ports? Generally speaking I'm under the impression if the host upon which login-shield is running, initiates the connection, it's not subject to ipset/iptables rules, which generally affect inbound connections, not outbound ones, but I could be wrong. Note that login-shield only restricts specified ports (unless you modify the script to block all ports) so whatever port LetsEncrypt uses could be whitelisted.

SomePersonSomeWhereInTheWorld commented 3 years ago

As far as the division by zero, that's probably because of a lack of data on file... just be patient and once you get some positive/negative entries in the log files, it should go away. (I didn't make the system so sophisticated it could avoid those errors right now. I wanted to code it mostly in shell script but a future version will probably be in Python and not have those glitches)

Yes within a few seconds the error went away, just a FYI, some simple error checking will fix.

For persistence after a reboot would it be as simple as running these scripts in order? Note I'm omitting 2:

yes, that works. The reason I didn't include details or intentionally make it persistent is I like the idea of "turning it on" intentionally, so you can really see how much difference it makes. When people are confident in the system, they can make it persistent. Plus, I don't reboot my servers that much.

I haven't tested but do the scripts accept a -y?

Now back to my other question:

How do you whitelist or remove a single IP address?

I have 2 professors logging in from 2 address in Bangladesh, 103.84.158.106 and 103.21.40.54, is there a whitelist mechanism?

 ./testip.sh 103.84.158.106

## Testing IP: 103.84.158.106 against IP blacklists...
Warning: 103.84.158.106 is in set login-shield.

For now I commented out #103.0.0.0/8 in ipset-main-nonUS.lst, ran./blacklist-main-nonUS.sh del and then ./blacklist-main-nonUS.sh.

Just FYI, I was an early, principal contributor to the Spamcop system. We had a difference of opinion when they allowed people to pay them to not be in their blacklist. So I began developing my own systems.

Lots of FP's with Spamcop earlier, was a very aggressive RBL. And yeah money tends to muck up lots of things.

SpareSimian commented 3 years ago

For the HTTP-based ACME protocol to renew one's certificate, the certbot script temporarily replaces one's web server with a simple one that serves a file from a temporary webroot that contains a digital hash. The Letsencrypt server pulls this file to verify the identity of the organization making the request. The address of their server changes due to load balancing, so it might fall in some cloud service block that your list includes. It looks like I could temporarily disable your rule in pre/post hooks that certbot invokes.

https://letsencrypt.org/docs/integration-guide/#firewall-configuration

DPsystems commented 3 years ago

As far as the division by zero, that's probably because of a lack of data on file... just be patient and once you get some positive/negative entries in the log files, it should go away. (I didn't make the system so sophisticated it could avoid those errors right now. I wanted to code it mostly in shell script but a future version will probably be in Python and not have those glitches)

Yes within a few seconds the error went away, just a FYI, some simple error checking will fix.

I'm aware of this. My plan is to re-write all of this in python soon.

I haven't tested but do the scripts accept a -y?

Hmm, no they don't. But the only one I believe that prompts to continue is the one that sets iptables.. and you could change that command to be persistent IIRC.

Personally, I like it not being automatically enabled upon reboot. If, for example, you accidentally get locked out, a reboot will reverse that.

How do you whitelist or remove a single IP address?

I have 2 professors logging in from 2 address in Bangladesh, 103.84.158.106 and 103.21.40.54, is there a whitelist mechanism?

There is not a whitelist mechanism within my system. I would recommend the best approach to this would be to add a specific rule to whitelist after the blacklist (since I believe iptables puts later commands higher up in precedence). I haven't tested this, but I assume that would work.

./testip.sh 103.84.158.106

Testing IP: 103.84.158.106 against IP blacklists...

Warning: 103.84.158.106 is in set login-shield.

For now I commented out #103.0.0.0/8 in ipset-main-nonUS.lst, ran./blacklist-main-nonUS.sh del and then ./blacklist-main-nonUS.sh.

cp ipset-main-nonUS.lst ipset-main-nonUS.me

then edit the .me version and comment it out. The script will look for a .me file first and supercede the .lst. This way you can continue to git pull new versions and decide if you want to make changes without having to comment it out every time.

Thanks for all the feedback. To be honest, I've been using this system for a long time and wasn't sure if anybody else would find it useful.

DPsystems commented 3 years ago

For the HTTP-based ACME protocol to renew one's certificate, the certbot script temporarily replaces one's web server with a simple one that serves a file from a temporary webroot that contains a digital hash. The Letsencrypt server pulls this file to verify the identity of the organization making the request. The address of their server changes due to load balancing, so it might fall in some cloud service block that your list includes. It looks like I could temporarily disable your rule in pre/post hooks that certbot invokes.

https://letsencrypt.org/docs/integration-guide/#firewall-configuration

Hmmmmm, I can't imagine login-shield would affect Certbot. Unless you're adding port 80/443 to the blocklist. I don't do that and don't recommend it because there's lots of ISP space in the blacklist. (The idea is, should anybody in China be trying to log into your IMAP or SSH? I don't think so, so I can block large swaths.. however, it may be legit to allow them access to the web so I don't block it - so I'm thinking this wouldn't interfere with certificate renewals).

btw, I have LetsEncrypt on six servers that are running login-shield and have never had an issue with auto renewing.

However, this may be something I might need to pay attention to for an upcoming web-based version I am working on (called web-shield). I'm particularly excited about this next set of scripts because it is proving to be VERY effective in stopping sql injection and web vulnerability probes. I have been monitoring logs now for more than a year and have a really nice blacklist I'm about to test on a larger scale - this does block web access to hosting platforms all around the world, including the US. I obviously am careful about not blocking important robots like Google, but I wonder if LetsEncrypt would be affected? We'll have to check, but still.. if the host initiates the connection, does it even matter? I'm under the impression ipset only comes into play for un-invited outside connections coming in?

SomePersonSomeWhereInTheWorld commented 3 years ago

How come after I run set-iptables del so I still see the rules in iptables?

./set-iptables.sh del
###### Running: set-iptables.sh
#
######
#
#  This script will DISABLE the LOGIN-SHIELD IP blacklist via IPTABLES
#
# WARNING:  This can cause you to lose connectivity to your server if not properly configured!
#
Continue (Y/n)?y
Yes
DELETING IPTABLES rules using ipset blacklist: login-shield for ports: 20,21,22,110,143,587,989,990,993,994,995,4190
iptables -D INPUT -p tcp --match multiport --dports 20,21,22,110,143,587,989,990,993,994,995,4190 -m set --match-set login-shield src -j DROP
iptables -D INPUT -p tcp --match multiport --dports 20,21,22,110,143,587,989,990,993,994,995,4190 -m set --match-set login-shield src -j LOG
iptables: No chain/target/match by that name.
## Done.
(base) [root@storm Login-Shield]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
LOG        tcp  --  anywhere             anywhere             multiport dports ftp-data,ftp,ssh,pop3,imap,submission,ftps-data,ftps,imaps,994,pop3s,sieve match-set login-shield src LOG level warning prefix "ShD-Lgn"
LOG        tcp  --  anywhere             anywhere             multiport dports ftp-data,ftp,ssh,pop3,imap,submission,ftps-data,ftps,imaps,994,pop3s,sieve match-set login-shield src LOG level warning prefix "ShD-Lgn"
LOG        tcp  --  anywhere             anywhere             multiport dports ftp-data,ftp,ssh,pop3,imap,submission,ftps-data,ftps,imaps,994,pop3s,sieve match-set login-shield src LOG level warning prefix "ShD-Lgn"
DROP       tcp  --  anywhere             anywhere             multiport dports ftp-data,ftp,ssh,pop3,imap,submission,ftps-data,ftps,imaps,994,pop3s,sieve match-set login-shield src
DROP       all  --  anywhere             anywhere             match-set blacklist-ip4 src
SpareSimian commented 3 years ago

LetsEncrypt would be affected by blocked ports 80 and 443 because of the way the ACME protocol works. The local host sets up a temporary web server, asks the LetsEncrypt server for renewal, and then the LetsEncrypt server contacts the local host on port 80 to verify that the requesting host owns that URL and is the one making the request. LetsEncrypt uses Akamei for this and that's what causes the uncertainty in the block list. We can't predict which Akamei server will connect to us.

DPsystems commented 3 years ago

How come after I run set-iptables del so I still see the rules in iptables?

The only time I see this happening is if the script parameters have been changed after the command was run. The iptables delete command will only work if the exact parameters match the version that's in the shell script. I've seen instances where I tweaked the .sh command and then it won't delete it because the two commands are slightly different. I outline this in the instructions - if you make any changes to the script, like change ports, you may have to manually delete the rules. But this could also be a variation in how the command works on different OS/versions. I'm using CentOS 6 as my test platform.

I also think in the future, more people will probably be using firewalld or some other command that "iptables" - I may have to update this, and I'm sure the commands may vary from one OS to another. This is something I may relay on others who have different systems to let me know if anything behaves differently on a different flavor of Linux.

DPsystems commented 3 years ago

LetsEncrypt would be affected by blocked ports 80 and 443 because of the way the ACME protocol works. The local host sets up a temporary web server, asks the LetsEncrypt server for renewal, and then the LetsEncrypt server contacts the local host on port 80 to verify that the requesting host owns that URL and is the one making the request. LetsEncrypt uses Akamei for this and that's what causes the uncertainty in the block list. We can't predict which Akamei server will connect to us.

So does this mean the whole web server is offline during cert renewal processes? I can't imagine that would be the case, but does the certbot renew command actually take Apache down, set it's own web server up, then bring Apache back up? If so, then yea, that could be an issue.

Is there a way to have any idea what IP ranges their systems may be running off of? I can make sure they're excluded. There are some IP ranges I just don't blacklist, like AWS because it's too difficult to target bad actors in that IP space (that's where Fail2Ban picks up the slack), so if they're using AWS, it's probably not an issue.

SpareSimian commented 3 years ago

If you look at /var/log/letsencrypt/letsencrypt.log, you can see that it does indeed take your Apache or Nginx server down and brings it back up with a custom config for each name in your cert.

According to the link above, they use Akamei content distribution servers, and I'm betting you exempt those already.

DPsystems commented 3 years ago

If you find an IP range for those servers, let me know - if you look in some of the .lst files, I will list some ranges there that I try to avoid blacklisting

SomePersonSomeWhereInTheWorld commented 3 years ago

I also think in the future, more people will probably be using firewalld or some other command that "iptables" - I may have to update this, and I'm sure the commands may vary from one OS to another. This is something I may relay on others who have different systems to let me know if anything behaves differently on a different flavor of Linux.

Definitely have to think about firewalld and that's even being replaced on Fedora with nftables.

Even Fail2ban had to be updated to use firewalld and rich-rules.

SomePersonSomeWhereInTheWorld commented 3 years ago

I'm trying to remove some of the IP ranges. I ran iptables -F but does ipsetneed to be cleared too? I still see:

temp.txt:Nov 13 11:50:59 ourserver kernel: ShD-LgnIN=bond0 OUT= MAC=e4:43:4b:76:b0:21:70:df:2f:43:d3:4d:08:00 SRC=177.182.160.210 DST=192.168.1.1 LEN=52 TOS=0x00 PREC=0x00 TTL=114 ID=13405 DF PROTO=TCP SPT=64268 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0

SomePersonSomeWhereInTheWorld commented 3 years ago

OK I think I see it now. Might want to include this in the instructions some where. ipset del login-shield 191.0.0.0/8 worked.

SpareSimian commented 3 years ago

Note that firewalld is a front end to other firewall kernel firewall systems. It can use iptables or nftables under the hood, and has its own persistence system for ipset. I'm using the firewalld-ipset back end for fail2ban.

SomePersonSomeWhereInTheWorld commented 3 years ago

Typo in INSTALL page: different shells scripts to

DPsystems commented 3 years ago

fixed the typo

DPsystems commented 3 years ago

If anybody has alternate versions of my set-iptables script for other OSes that use nftables, let me know - I would like to make this compatible with that.

SomePersonSomeWhereInTheWorld commented 3 years ago

I've had to remove several IP ranges from students and professors who were blocked by the ranges. I know you mentioned there's not white list feature but is there a(n) (easy) way to edit the class A/8 ranges to accommodate these users without opening the entire range?

Would you like to know the IP address (offline) that were from real users? I suppose these IPs could still be part of subnets that have been considered malicious before.

Anyways consider a whitelist feature as a suggestion.

DPsystems commented 3 years ago

Yes, feel free to e-mail me specifics on the IP ranges. I do want to know which blocks may be troublesome.

Although I freely admit I fully expect this to happen, especially if you use the US blocklist and you're in the US.

There are two ways you can address this situation currently, which are outlines in the docs I believe. I think the best way would be to create your own whitelist that supercedes the blacklist (or I guess, is loaded AFTER login-shield using IPSET) that's probably the "industry best practice" anyway: specifically whitelisting the IP space you need to grant login access to.

The other way is to remove the offending rule from the .lst file and save it to a .me extension and it will be used instead of the .lst. Then if you update, you may have to diff the files and make revisions if you desire.

However, you do bring up a good point... I guess I should be able to create some sort of whitelist.me file that contains your allowed addresses. I'm not a super expert on iptables, but if I load a command that allows connections from an ipset list, will that take precedence over a later iptables rule that blocks them? I haven't tried it.. maybe someone can verify that's doable? If so, then I can simply add some more code to create an ipset whitelist list, and put code in the set-iptables.sh script to run that command last and specifically allow whitelisted CIDRs. Can anybody else confirm this is the way to go? If so I can get on that for the next major revision.

SomePersonSomeWhereInTheWorld commented 3 years ago

I have a user using a VPN ExpressVPN and the following IPs are being blocked: 45.146.55.166 45.41.132.12

However I only see 1 in our logs:

ShD-LgnIN=eno3 OUT= MAC=XX:XX:XX:XX SRC=45.146.55.177 DST=XX.XX.XX.XX LEN=64 TOS=0x00 PREC=0x00 TTL=54 ID=0 DF PROTO=TCP SPT=59400 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0 This IP is not in any of the ipset-* files. Any idea how to fix this?

DPsystems commented 3 years ago

45.146.164.0/23 is in webblock-nonUS.lst

The other one isn't in our list. I would suspect that the report that IP is blocked is anomalous.

You can make a copy of webblock-nonUS.lst and rename it to webblock-nonUS.me with the IP you want to remove and then re run things. (or run ./blacklist-US-hosting.sh del before creating the .me version, then re run it without the "del" parameter)

SomePersonSomeWhereInTheWorld commented 3 years ago

45.146.164.0/23 is in webblock-nonUS.lst

Do you mean ipset-main-nonUS.lst

I don't see it there.

You can make a copy of webblock-nonUS.lst and rename it to webblock-nonUS.me with the IP you want to remove and then re run things. (or run ./blacklist-US-hosting.sh del before creating the .me version, then re run it without the "del" parameter)

So after a reboot you'd run the .me file? Dow do you merge any change you make?

DPsystems commented 3 years ago

Oh I'm sorry, wrong file. That's part of an upcoming project.

The offending line is in ipset-others.lst and is this: 45.128.0.0/11

It would be nice if ipset would list the actual rule that targets the IP. You can run "testip" or ipset test login-shield to see if it's in the list, but then sometimes it's a bit of a chase to figure out the specific rule

So the way it works is, when you go to set the blacklist, there are shell scripts for each list. When the script runs it will look for a .me version of the .lst lists. So you can make a copy of ipset-others.lst and remove the offending line and save it as ipset-others.me. Then the next time you run the script it will favor the .me list over the .lst file.

You could also create your own shell script that runs after you set up login-shield that un-blacklists the IP ranges you want like this:

ipset del login-shield 45.128.0.0/11

SomePersonSomeWhereInTheWorld commented 3 years ago

OK very helpful tips.

Now any idea how this log was created?

ShD-LgnIN=eno3 OUT= MAC=XX:XX:XX:XX SRC=45.146.55.177 DST=XX.XX.XX.XX LEN=64 TOS=0x00 PREC=0x00 TTL=54 ID=0 DF PROTO=TCP SPT=59400 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0

It clearly shows ShD-LgnIN but that IP doesn't appear to be in any of the list ranges. At least now from what I cold discern.

DPsystems commented 3 years ago

Ok, I found the rule affecting that. It's this range: 45.128.0.0/11 in ipset-others.lst

What I do to track these things down is run a grep (ip range) *.lst in the login-shield directory and look for similar rules. I found it using this:

grep 45.1 *.lst which revealed:

ipset-others.lst:45.14.148.0/22 ipset-others.lst:45.119.0.0/16 ipset-others.lst:45.160.0.0/11 ipset-others.lst:45.128.0.0/11 ipset-others.lst:45.192.0.0/12 ipset-others.lst:45.116.0.0/16 ipset-others.lst:45.118.0.0/16 ipset-others.lst:45.160.0.0/11 ipset-others.lst:45.120.0.0/16 ipset-others.lst:45.10.40.0/24 ipset-others.lst:45.10.41.0/24 ipset-others.lst:45.10.42.0/24 ipset-others.lst:45.10.43.0/24 ipset-others.lst:45.117.0.0/16 ipset-others.lst:45.114.0.0/16 ipset-others.lst:45.115.0.0/16 ipset-proxies.lst:45.152.180.0/24

Then I use a CIDR calculator to check various ranges to see if the offending address is there https://www.ipaddressguide.com/cidr

NOTE that I do tend to target a reasonable amount of VPN IP space because it's such a source of malicious activity.

You might be better off, specifically white-listing any clients who use VPNs. It would be safer than allowing a large block of rogue space the ability to brute force your logins (although I might also assume you're using Fail2Ban which is a good fallback)

DP