search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
9.03k stars 467 forks source link

Bug/ decision list missing IP's #1572

Closed srulikuk closed 1 year ago

srulikuk commented 2 years ago

ClearOS 7 (CentOS 7) crowdsec version: v1.3.4-el7-rpm

I am testing crowdsec at the moment and have found that some banned IP's do not appear in the decision list however they are in my ipset.

the syslog file shows 21 attempts on the same timestamp, crowdsec.log shows the detection and bans however cscli decision list does not show this IP however when I test the ipset its listed there.

Below are the outputs of the syslog, crowdsec.log and ipset test.

syslog:

Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1021 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1022 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1023 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1024 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1025 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1026 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1027 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1028 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=1029 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=5085 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=9050 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=15060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=45060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=55060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=65060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=25060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=35060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=9060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=5090 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=8060 
Jun  3 01:23:04 gateway kernel: Closed port probe: IN=ppp0 SRC=45.95.147.12 PROTO=UDP DPT=5080

crowdsec.log:

time="03-06-2022 01:23:04" level=info msg="Ip 45.95.147.12 performed 'crowdsecurity/iptables-closed-ports_udp' (11 events over 11.493911ms) at 2022-06-03 00:23:04.917542197 +0000 UTC"
time="03-06-2022 01:23:04" level=info msg="Ip 45.95.147.12 performed 'crowdsecurity/iptables-closed-ports_udp' (11 events over 10.537075ms) at 2022-06-03 00:23:04.92940284 +0000 UTC"
time="03-06-2022 01:23:04" level=info msg="Ip 45.95.147.12 performed 'crowdsecurity/iptables-closed-ports_udp' (11 events over 9.882617ms) at 2022-06-03 00:23:04.940582277 +0000 UTC"
time="03-06-2022 01:23:05" level=info msg="(d5244d56bb9546688afd1bcc1db9e37aPBgP7VuTRLpO4Qw7/crowdsec) crowdsecurity/iptables-closed-ports_udp by ip 45.95.147.12 (NL/49870) : 24h ban on Ip 45.95.147.12"
time="03-06-2022 01:23:05" level=info msg="(d5244d56bb9546688afd1bcc1db9e37aPBgP7VuTRLpO4Qw7/crowdsec) crowdsecurity/iptables-closed-ports_udp by ip 45.95.147.12 (NL/49870) : 24h ban on Ip 45.95.147.12"
time="03-06-2022 01:23:05" level=info msg="(d5244d56bb9546688afd1bcc1db9e37aPBgP7VuTRLpO4Qw7/crowdsec) crowdsecurity/iptables-closed-ports_udp by ip 45.95.147.12 (NL/49870) : 24h ban on Ip 45.95.147.12"
time="03-06-2022 01:23:05" level=info msg="(d5244d56bb9546688afd1bcc1db9e37aPBgP7VuTRLpO4Qw7/crowdsec) crowdsecurity/iptables-closed-ports_udp by ip 45.95.147.12 (NL/49870) : 24h ban on Ip 45.95.147.12"

IPSET:

# ipset test crowdsec-blacklists 45.95.147.12
Warning: 45.95.147.12 is in set crowdsec-blacklists

I am using my own collection which is similar to crowdsec iptables-port-scanner my collection is here https://github.com/srulikuk/crowdsec

blotus commented 2 years ago

Hello,

It looks like you hit a weird corner case: 45.95.147.12 is part of the community blocklist (you should see the ip if you run cscli decisions list -a | grep 45.95.147.12). It does not show in cscli decisions list because we explicitly ignore IP that belongs to the community blocklist.

The IP should have been blocked by the bouncer, are you using the firewall bouncer in ipset mode (ie, the bouncer manages only the set content and you created the iptables rule) ?

If yes, there is probably something wrong with the rule you added. If no, this could be an issue with the bouncer or another rule allowing the traffic before the bouncer rule drops it.

srulikuk commented 2 years ago

Seems a weird case indeed

(you should see the ip if you run cscli decisions list -a | grep 45.95.147.12).

# cscli decisions list -a | grep 45.95.147.12

| 803038 | crowdsec | Ip:45.95.147.12 | crowdsecurity/iptables-closed-ports_udp | ban | NL | 49870 Alsycon B.V.  | 11 | 5h55m59.952899847s |  670 |

without the -a there are no results and if I understood you correctly the -a should not show my scenario as the detector for the ban, furthermore if its in the community blocklist how is it that it wasn't in my ipset blocklist to start with.

With regards to my bouncer I am using it out of the box, just installed and it seems to work; iptables -nvL

19424  815K DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set crowdsec-blacklists src

my crowdsec-firewall-bouncer.log has 1,000's of entries like this

time="27-05-2022 02:17:26" level=info msg="2800 decisions added"
time="27-05-2022 02:17:37" level=info msg="2700 decisions added"
time="27-05-2022 02:17:47" level=info msg="2491 decisions added"

the ipset has just over 13.5k IP's so it seems to work correctly.

This does not see like a corner case however, if i grep the decision list for my scenario name;

# cscli decision list | grep "iptables-closed-ports_" | wc -l
93
# cscli decision list -a | grep "iptables-closed-ports_" | wc -l
313

So unless I am the one feeding these IP's to the community blocklist (which I highly doubt given I am only running for a few days) this is weird indeed.

blotus commented 2 years ago

Can you give me the full output of iptables -L ?

It seems there are 2 issues here:

srulikuk commented 2 years ago

Hi, Below is the output of my iptables, just a little background, this machine is a router, it sits on my wan and does port forwarding (NAT) local.ip in the output is the local ip of the router. ( i also included the output of the iptables nat table below)

# iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists src
DROP       all  --  anywhere             anywhere             match-set snortsam_INGRESS src
DROP       all  --  anywhere             anywhere             state INVALID
REJECT     tcp  --  anywhere             anywhere             tcp flags:SYN,ACK/SYN,ACK state NEW reject-with tcp-reset
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN state NEW
DROP       all  --  loopback/8           anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply
ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:bootps dpt:bootpc
ACCEPT     tcp  --  anywhere             local.ip           tcp dpt:icp
ACCEPT     tcp  --  anywhere             local.ip           tcp dpt:81
ACCEPT     udp  --  anywhere             anywhere             udp dpts:1024:65535 state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:1024:65535 state RELATED,ESTABLISHED
LOG        tcp  --  anywhere             anywhere             state NEW multiport dports  !icp,netiq,rockwell-csp1,31222,44393,44395,81 /* RULE # 1 of 1 port probing LOG excluding -> 1112,2220,2221,31222,44393,44395,81 */ LOG level warning prefix "Closed port probe: "
LOG        udp  --  anywhere             anywhere             state NEW /* RULE # 1 of 1 port probing LOG excluding ->  */ LOG level warning prefix "Closed port probe: "

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set snortsam_SELF src,dst,dst
DROP       all  --  anywhere             anywhere             match-set snortsam_EGRESS dst
DROP       all  --  anywhere             anywhere             match-set snortsam_INGRESS src
ACCEPT     tcp  --  anywhere             192.168.10.110       tcp dpt:44393
ACCEPT     tcp  --  anywhere             192.168.10.110       tcp dpt:44395
ACCEPT     tcp  --  anywhere             192.168.10.149       tcp dpt:31222
ACCEPT     tcp  --  anywhere             192.168.10.110       tcp dpt:rockwell-csp1
ACCEPT     tcp  --  anywhere             192.168.10.120       tcp dpt:netiq
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set snortsam_SELF src,dst,dst
DROP       all  --  anywhere             anywhere             match-set snortsam_EGRESS dst
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootpc dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:bootpc dpt:bootps
ACCEPT     tcp  --  local.ip             anywhere             tcp spt:icp
ACCEPT     tcp  --  local.ip             anywhere             tcp spt:81
ACCEPT     all  --  anywhere             anywhere            

Chain DROP-lan (0 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             local.ip           tcp dpt:44393 to:192.168.10.110:44393
DNAT       tcp  --  anywhere             local.ip           tcp dpt:44395 to:192.168.10.110:44395
DNAT       tcp  --  anywhere             local.ip           tcp dpt:31222 to:192.168.10.149:31222
DNAT       tcp  --  anywhere             local.ip           tcp dpt:rockwell-csp1 to:192.168.10.110:2221
DNAT       tcp  --  anywhere             local.ip             tcp dpt:netiq to:192.168.10.120:2220

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
SNAT       tcp  --  192.168.10.0/24     192.168.10.110    tcp dpt:44393 to:192.168.10.100
SNAT       tcp  --  192.168.10.0/24     192.168.10.110    tcp dpt:44395 to:192.168.10.100
SNAT       tcp  --  192.168.10.0/24     192.168.10.149    tcp dpt:31222 to:192.168.10.100
SNAT       tcp  --  192.168.10.0/24     192.168.10.110    tcp dpt:rockwell-csp1 to:192.168.10.100
SNAT       tcp  --  192.168.10.0/24     192.168.10.120         tcp dpt:netiq to:192.168.10.100
MASQUERADE  all  --  anywhere             anywhere
srulikuk commented 2 years ago

@blotus any ideas? also my crowdsec-blacklists has around 12.5k ip's is that normal? seems on the low side to me. thanks

LaurenceJJones commented 2 years ago

Hey! I seen this issue been outstanding for some time: 12.5k IP's can be normal due to the fact that the CAPI blocklist you receive is tailored to your installed scenarios. Are you still experiencing issues with our rule not blocking?

LaurenceJJones commented 1 year ago

Closing issue due to no response from OP, reopen if you still experiencing an issue