search

StamusNetworks / SELKS

A Suricata based IDS/IPS/NSM distro
https://www.stamus-networks.com/open-source/#selks
GNU General Public License v3.0
1.43k stars 284 forks source link

Suricata IPREP rules #289

Closed ManuelFFF closed 3 years ago

ManuelFFF commented 3 years ago

Hi,

Note: I am running SELKS 6 up to date.

Would it be possible to get some help on configuring Suricata as IPS to block all traffic to/from bad IP addresses from a black list file? I have been researching for a while and it looks like "IP Reputation" mechanism could do the work, but I have not been able to do it on my own.

This is what I have tried so far:

Thank you in advance

regit commented 3 years ago

You need to use 2600000 as category name in the signature and not 10.

Side note, it will be more simple to use a dataset for this signature as what you are checking is the presence of the IP in a list.

ManuelFFF commented 3 years ago

Hi @regit ,

Thank you for the advice. I will try the new settings in a bit. Would you mind to elaborate a little bit more about the other way that you suggest?

Thank you

ManuelFFF commented 3 years ago

Every so often the custom line that I added to /etc/suricata/rules/scirius-categories.txt is removed. Is there a process checking the integrity of this file or overwriting it?

ManuelFFF commented 3 years ago

Hi @regit ,

I got the same error after applied the new settings. I restarted Suricata and Scirius before trying again.

scirius_custom_source

ManuelFFF commented 3 years ago

Hi @regit ,

I tried adding the new rule/signature via CLI directly to Suricata and I did not get any errors, but I keep getting the same error if I try via Scirius. Am I doing something wrong or this might be a bug in Scirius. Please help.

Thank you

pevma commented 3 years ago

I just tested that too the same way as you and there were no erros. I think what you need to do is create tar.gz file that has inside a rules folder and inside the rules folder :

ManuelFFF commented 3 years ago

Oh I just noticed that my custom category (last line) disappear from file /etc/suricata/rules/scirius-categories.txt every time I update Suricata via Scirius (Suricata Ruleset Actions).

ManuelFFF commented 3 years ago

Hi @pevma,

Can I have more than one category file in Suricata? I saw that you can have several iprep files and rules file, but it looks like only one category file can be active each time. Pleas correct me if I am wrong.

I fact I started this way:

I have been trying the above in SELKS 6 with Suricata v6.0.0 and SELKS 6 with Suricata v7.0.0

If you see that I am missing something, any details, please let me know. I would love to continue managing Suricata via Scirius

Thank you

pevma commented 3 years ago

You can take the category and add it in as we as i explained ?

ManuelFFF commented 3 years ago

You mean referencing additional config file within suricata.yaml, so I can add a custom category file in the custom config file?

##
## Include other configs
##

# Includes:  Files included here will be handled as if they were in-lined
# in this configuration file. Files with relative pathnames will be
# searched for in the same directory as this configuration file. You may
# use absolute pathnames too.
# You can specify more than 2 configuration files, if needed.
#include: include1.yaml
#include: include2.yaml
include: /etc/suricata/selks6-addin.yaml
include: /etc/suricata/test-addin.yaml
pevma commented 3 years ago

No I meant including the category file in the tar.

-- Regards, Peter Manev

On 14 Jan 2021, at 22:46, ManuelFFF notifications@github.com wrote:

 You mean referencing additional config file within suricata.yaml, so I can add a custom category file in the custom config file?

Include other configs

Includes: Files included here will be handled as if they were in-lined

in this configuration file. Files with relative pathnames will be

searched for in the same directory as this configuration file. You may

use absolute pathnames too.

You can specify more than 2 configuration files, if needed.

include: include1.yaml

include: include2.yaml

include: /etc/suricata/selks6-addin.yaml include: /etc/suricata/test-addin.yaml — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

ManuelFFF commented 3 years ago

I'll try that out

ManuelFFF commented 3 years ago

This is great! This time Scirius processed the .tar.gz without issues. Upon applying Suricata Ruleset Actions, the new category was added to same file /etc/suricata/rules/scirius-categories.txt. Can I continue using my custom reputation file or even an online source of blacklist IPs?

Thank you

pevma commented 3 years ago

Yes you can.

ManuelFFF commented 3 years ago

Hi @regit and @pevma ,

I was wondering if I can count on you so I can keep moving forward on my Suricata custom config as IPS.

My Suricata will be in gateway mode, protecting the computers behind Suricata.

There are some concepts and configurations that are not entirely clear to me or I do not feel confident after having reviewed a bit of documentation here and there. Can you please provide some light?:

  1. Inline does not mean the same as IPS. Does this refer to inline with IDS mode, just sniffing traffic and alerting, but not dropping packets?
  2. Can IPS mode be enabled with only one network port or do I always need at least 2 network cards?
  3. On a computer with enough network ports, should I have 2 instances of Suricata running, one as IDS and the other as IPS, or can I have the same instance running both modes? Let's say eth0and eth1for IPS and eth2for IDS. I found this article from Regit: "... Please note that is possible to have normal IDS interface running simultaneously. For example, eth3 could be added to the af-packet configuration and used a regular interface..."
  4. To run Suricata in IPS mode and actively block some packets, do I need to involve / configure the firewall (iptables) or this is optional? I have seen configs like this: 
     sudo iptables -I INPUT -j NFQUEUE
 sudo iptables -I OUTPUT -j NFQUEUE
  5. Do I need to set a logical bridge for Suricata to work in IPS mode?
  6. Will any of this breake SELKS workflow?

I probably have more questions, but I think these are enough for now.

Thank you in advance

ManuelFFF commented 3 years ago

Hi,

I would appreciate if you could answer me some of the questions raised in my previous post. At least points # 3, # 4 and # 5.

Thank you in advance

ManuelFFF commented 3 years ago

Sorry, my browser was acting funny and I closed this thread by mistake. I still need help. Thank you

pevma commented 3 years ago

:) Browser issues

There are two approaches you can take - described here https://suricata.readthedocs.io/en/suricata-6.0.0/setting-up-ipsinline-for-linux.html

For af_packet with SELKS you can also use that https://github.com/StamusNetworks/SELKS/wiki/Initial-Setup---Suricata-IPS

ManuelFFF commented 3 years ago

Thanks Pevma. I'll take a look.

ManuelFFF commented 3 years ago

So far I can see that definitely I do need to set the firewall in order to use Surcata in IPS mode. I wasn't using iptables at all, so my first step will be enable iptables, well I think actually nftables replaced iptables framework starting Debian 10. Researching and trying to set basic rules.

In the docs I am supposed to use this command to run Suricata with NFQUEUE

sudo suricata -c /etc/suricata/suricata.yaml -q 0

But the above command will leave the process running on a console window that I can't close, or this will end the process. Instead I think that I need to modify the Suricata service or the script the service calls, but I would not know what to change. Can you help me?

Script is located in /etc/init.d/suricata

And this is the service

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/suricata start
ExecStop=/etc/init.d/suricata stop
pevma commented 3 years ago

You can use af-packet mode or nfqueue, either or , depending on your needs. AFP is more like a bridge.

The /etc/init.d/suricata is part of the regular suricata package. There is an example of systemd in the code here that you can use https://github.com/OISF/suricata/tree/master/etc

ManuelFFF commented 3 years ago

Hi,

I could notice AFP works like an internal bridge within Suricata, copying packets from network interface #1 to #2. So is there no need to config a logical bridge in the OS hosting Suricata?

Another question regarding this article: Do I have to choose between af-packets and nfqueuemodes/settings? I interpreted that all steps described in the article were part of the same config, like I would need to configure nfqueue(so all traffic it's being redirected to the queue Suricata can read/handle) and also the af-packets (so Suricata can read packets from one network interface, apply rules, and send them to the other network interface).

Please correct me if I'm wrong as I'm still learning.

As for the service modification, do you think I can use the following?

ExecStart=/sbin/suricata -c /etc/suricata/suricata.yaml -q 0 --pidfile @e_rundir@suricata.pid
ExecStop=/bin/kill -USR2 $MAINPID

As always, thank you for your time and help

ManuelFFF commented 3 years ago

I think I understand this a little bit more (please correct me if I am wrong). I have to choose a layer 3 OR a layer 2 configuration, but NOT both NOR combine them in any way. Also if I choose layer 3, I have to use only one of the three choices available.

1. Suricata in layer3 This includes either of the following options, but NOT a combination of them:

2. Suricata in layer 2 This includes only one option:

Does the above make sense?

Thank you

ManuelFFF commented 3 years ago

Thank you

pevma commented 3 years ago

Yes, one is L3 the other L2. With IPS in AFP you need to use AFPv2 - as given an example - https://github.com/StamusNetworks/SELKS/wiki/Initial-Setup---Suricata-IPS I suggest you can start with cluster type cluster_flow. You can leave defrag to yes.

ManuelFFF commented 3 years ago

Hi,

ExecStart=/sbin/suricata -c /etc/suricata/suricata.yaml -q 0 --pidfile @e_rundir@suricata.pid
ExecStop=/bin/kill -USR2 $MAINPID

Thank you

pevma commented 3 years ago
  1. Both L2/L3 have pros and cons, depends on your needs actually. I usually do AFP but have done both
  2. No you are all set , no extra packages needed.
  3. the main config is fine
  4. it is better to have 3
  5. this looks like you will be using L3 approach. You can also just look and enable settings in /etc/default/suricata. Those are picked up by /etc/init.d/suricata
ManuelFFF commented 3 years ago

Hi,

I appreciate all your answers. I'll be trying L2 approaching first, so I think editing the service won't be necessary for now.

Thank you

ManuelFFF commented 3 years ago

Hi,

I have been trying to use the settings as proposed here. I have a question regarding the network config. Could you please explain the purpose of the following lines? Are those strictly necessary?

allow-hotplug eno1
iface eno1 inet static  # Management interface
address 192.168.0.100
netmask 255.255.252.0
gateway 192.168.0.1
dns-nameservers 192.168.0.11 192.168.0.6

iface enp1s0f0 inet manual
     pre-up ifconfig $IFACE up
     post-down ifconfig $IFACE down
     post-up /etc/network/if-up.d/selks-idps-interface-tuneup_stamus

iface enp1s0f1 inet manual
     pre-up ifconfig $IFACE up
     post-down ifconfig $IFACE down
     post-up /etc/network/if-up.d/selks-idps-interface-tuneup_stamus

Since you told me that it was recommended at least 3 network interfaces to run Suricata in IPS mode (2 interfaces for af-packets bridge, 1 interface for management), then I added two usb-to-ethernet adapters to the test machine. New devices are identical and were detected by NetworkManager.

user1@server1:~$ sudo nmcli device
DEVICE           TYPE      STATE         CONNECTION
enp2s0           ethernet  connected     WiredCon1
enxa0cec8d92d70  ethernet  disconnected  --
enxa0cec8d92e2e  ethernet  disconnected  --
lo               loopback  unmanaged     --

user1@server1:~$ sudo nmcli connection show
NAME       UUID                                  TYPE      DEVICE
WiredCon1  5054ce93-d3dc-4f67-be40-4dfe94c16030  ethernet  enp2s0

After that I tried to update the config above to use the new hardware as follows:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

allow-hotplug WiredCon1
iface WiredCon1 inet static  # Management interface
address 192.168.1.150/24
#netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 8.8.8.8

iface enxa0cec8d92d70 inet manual
     pre-up ifconfig $IFACE up
     post-down ifconfig $IFACE down
     post-up /etc/network/if-up.d/selks-idps-interface-tuneup_stamus

iface enxa0cec8d92e2e inet manual
     pre-up ifconfig $IFACE up
     post-down ifconfig $IFACE down
     post-up /etc/network/if-up.d/selks-idps-interface-tuneup_stamus

WiredConfig1

[connection]
id=WiredCon1
uuid=2072879f-c920-3c52-bef3-8a31dd886a2c
type=802-3-ethernet

[802-3-ethernet]
mac=64:00:6a:22:d1:bb

[ipv4]
method=manual
dns=8.8.8.8;
addresses1=192.168.1.150;24;192.168.1.1;

[ipv6]
method=auto
ip6-privacy=2

WiredCon1.nmconnection

[connection]
id=WiredCon1
uuid=5054ce93-d3dc-4f67-be40-4dfe94c16030
type=ethernet
permissions=
timestamp=1611592251

[ethernet]
mac-address-blacklist=

[ipv4]
address1=192.168.1.150/24,192.168.1.1
dns=8.8.8.8;
dns-search=
method=manual

[ipv6]
addr-gen-mode=eui64
dns-search=
ip6-privacy=2
method=auto

If I go to the SELKS Desktop now I can see 3 "WiredCon1" available for connection, but only one connected. If I try to connect any of the other devices, it will success, but previous connection will be terminated. I am failing to have 3 network interfaces up for Suricata.

Please help me with this issue and to understand/fix what I am missing or doing wrong.

Thank you

ManuelFFF commented 3 years ago

Update

I removed the file manually added "WiredCon1" from /etc/NetworkManager/system-connections/ Restarted the machine System connected to main interface "enp2s0" as expected The other two interfaces are available as ifupdown, but not connected. Not even if I force them to connect

user1@server1:~$ sudo nmcli device
DEVICE           TYPE      STATE         CONNECTION
enp2s0           ethernet  connected     enp2s0
enxa0cec8d92d70  ethernet  disconnected  --
enxa0cec8d92e2e  ethernet  disconnected  --
lo               loopback  unmanaged     --

user1@server1:~$ sudo nmcli connection show
NAME                        UUID                                  TYPE      DEVICE
enp2s0                      d9866f0c-0ec6-4a5e-846f-099a08ac1e75  ethernet  enp2s0
Ifupdown (enxa0cec8d92d70)  bdedb4f9-7920-e7c6-8801-8ccab485e04f  ethernet  --
Ifupdown (enxa0cec8d92e2e)  19af9b73-8cf7-71f1-64a0-5318e5135851  ethernet  --
Wired connection 1          90484b20-8596-3d6b-923c-74e9b40384f9  ethernet  --
user1@server1:~$ sudo nano /etc/network/interfaces
user1@server1:~$ sudo nmcli device connect enxa0cec8d92d70
Error: Connection activation failed: (5) IP configuration could not be reserved (no available address, timeout, etc.).

net interfaces

ManuelFFF commented 3 years ago

Update

If I restart the machine, one of the new network interfaces is automatically connecting to "Wired Connection 1" (added automatically by the system). I also noticed that iface enp2s0seems to be duplicated in the connections. Is this the expected behavior? 

user1@server1:~$ sudo nmcli device
DEVICE           TYPE      STATE         CONNECTION
enxa0cec8d92e2e  ethernet  connected     Wired connection 1
enp2s0           ethernet  connected     enp2s0
enxa0cec8d92d70  ethernet  disconnected  --
lo               loopback  unmanaged     --

user1@server1:~$ sudo nmcli connection show
NAME                        UUID                                  TYPE      DEVICE
enp2s0                      1a95bb7e-9133-4229-8ac4-3a7b3e84856b  ethernet  enp2s0
Wired connection 1          0bed47ba-a4f3-3bbc-8912-13b23c8ceace  ethernet  enxa0cec8d92e2e
Ifupdown (enp2s0)           2072879f-c920-3c52-bef3-8a31dd886a2c  ethernet  --
Ifupdown (enxa0cec8d92d70)  bdedb4f9-7920-e7c6-8801-8ccab485e04f  ethernet  --

I was expecting to see either all three ifaces as Ifupdown:

Ifupdown (enp2s0)           2072879f-c920-3c52-bef3-8a31dd886a2c  ethernet  --
Ifupdown (enxa0cec8d92d70)  bdedb4f9-7920-e7c6-8801-8ccab485e04f  ethernet  --
Ifupdown (enxa0cec8d92e2e)  0bed47ba-a4f3-3bbc-8912-13b23c8ceace  ethernet --

or at least enp2s0connected either to enp2s0or Wired Connection 1 and the other ifaces as Ifupdown. I feel that I could be either missing something or doing something wrong...

enp2s0                                    2072879f-c920-3c52-bef3-8a31dd886a2c  ethernet  --
Ifupdown (enxa0cec8d92d70)  bdedb4f9-7920-e7c6-8801-8ccab485e04f  ethernet  --
Ifupdown (enxa0cec8d92e2e)  0bed47ba-a4f3-3bbc-8912-13b23c8ceace  ethernet --
pevma commented 3 years ago

The AFP IPS interface should just be set up as "bridging" no IP/iptables/nftables setup is needed. You could also consult https://suricata.readthedocs.io/en/suricata-6.0.1/setting-up-ipsinline-for-linux.html?highlight=ips#af-packet-ips-mode

ManuelFFF commented 3 years ago

Hi,

I think I have done all recommended in the article above.

Few notes:

user1@server1:~$ sudo nmcli device
DEVICE           TYPE      STATE         CONNECTION
enp2s0           ethernet  connected     Wired connection 1
enxa0cec8d92d70  ethernet  disconnected  --
enxa0cec8d92e2e  ethernet  disconnected  --
lo               loopback  unmanaged     --

user1@server1:~$ sudo nmcli connection show
NAME                        UUID                                  TYPE      DEVICE
Wired connection 1          90484b20-8596-3d6b-923c-74e9b40384f9  ethernet  enp2s0
Ifupdown (enxa0cec8d92d70)  bdedb4f9-7920-e7c6-8801-8ccab485e04f  ethernet  --
Ifupdown (enxa0cec8d92e2e)  19af9b73-8cf7-71f1-64a0-5318e5135851  ethernet  --

The above looks a little bit better, although I'm not sure if is the expected result.

Perhaps the article from SELKS was referring to /etc/suricata/selks6-interfaces-config.yaml when it mention /etc/suricata/selks5-addin.yaml at the beginning. I know it's an old doc that we are trying to re-use. Do you think all IPS settings should go there (custom SELKS file) rather that the Suricata main config file? If that is the case, Suricata main config file will have IDS settings for af-packet and SELKS custom file will have IPS settings for af-packet. Will this be an issue or what should go to each file?

Thank you

pevma commented 3 years ago

The configs are explained in the docs here - https://github.com/StamusNetworks/SELKS/wiki/Config-files Basically selks6 overwrites certain settings.

ManuelFFF commented 3 years ago

OK, so if I got it right, per the article you sent, it does not matter if the Suricata main config file is in IDS or IPS mode. SELKS custom config files selks6-addin.yaml and selks6-interfaces-config.yaml will overwrite any config included that match the main config file. That being said, I only need to edit SELKS files. Am I correct?

ManuelFFF commented 3 years ago

Another question. I noticed that in Suricata main config file, the other capture protocols (other than af-packet) have a different value for - interface: and cluster-id:. 

pfring:
  - interface: enp2s0
    cluster-id: 99
netmap:
 - interface: enp2s0
pcap:
  - interface: enp2s0

Should I add custom settings for those in SELKS custom files to overwrite settings and match new IPS settings? It would look like the following. Please let me know if I can use this config:

Note: Host 150 is where Suricata is running, so we don't want any capture/report including this machine. That is why I included the bpf-filter:

selks6-interfaces-config.yaml

af-packet:
  - interface: enxa0cec8d92d70
    threads: 1
    defrag: yes
    cluster-type: cluster_flow
    cluster-id: 98
    copy-mode: ips
    copy-iface: enxa0cec8d92e2e
    tpacket-v3: no
    ring-size: 2048
    buffer-size: 64535
    use-mmap: yes
    bpf-filter: not host 192.168.1.150
  - interface: enxa0cec8d92e2e
    threads: 1
    cluster-id: 97
    defrag: yes
    cluster-type: cluster_flow
    copy-mode: ips
    copy-iface: enxa0cec8d92d70
    tpacket-v3: no
    ring-size: 2048
    buffer-size: 64535
    use-mmap: yes
    bpf-filter: not host 192.168.1.150

pcap:
  - interface: enxa0cec8d92d70
    cluster-id: 98
    bpf-filter: "not host 192.168.1.150"
  - interface: enxa0cec8d92e2e
    cluster-id: 97
    bpf-filter: "not host 192.168.1.150"

pfring:
  - interface: enxa0cec8d92d70
    cluster-id: 98
    bpf-filter: not host 192.168.1.150
  - interface: enxa0cec8d92e2e
    cluster-id: 97
    bpf-filter: not host 192.168.1.150

netmap:
  - interface: enxa0cec8d92d70
    cluster-id: 98
    bpf-filter: not host 192.168.1.150
  - interface: enxa0cec8d92e2e
    cluster-id: 97
    bpf-filter: not host 192.168.1.150
pevma commented 3 years ago

Yes for AFP only since you are not using pfrng/netmap etc

ManuelFFF commented 3 years ago

Hi,

I have Suricata running in IPS mode (L2 configuration) already. I managed to acomodate all the network interfaces and the internal AFP bridge seems to be working fine, as I can access a test PC behind Suricata and from that PC I can reach Internet. After that I have been running tests. All rules are IP-Only using the iprep control. During these tests all other Suricata rules are disabled.

The last results showed me that not all the traffic from the reputation list it's being blocked. Do you have any idea about what could be failing or what should I check? If you need more details, let me know.

Please help!

Thanks

ManuelFFF commented 3 years ago

I have an update:

I believe the issue lies within the new categories/rules I have added later, but I can't find the root cause. I hope you can help me find it. I have noticed that rules matching manual iprep file test-iprep.list will work like charm. This file has only a few IPs matching only one category 30,TESTBadIP,TEST Known Bad IP Reputation.

I have added a few more categories and iprep files and signatures from blacklists downloaded from Internet. New categories and signatures were added via Scirius without issues.

/etc/suricata/rules/scirius-categories.txt

1,2400000,ET DROP Spamhaus DROP Listed Traffic Inbound
2,2402000,ET DROP Dshield Block Listed Source
3,2403300,ET CINS Active Threat Intelligence Poor Reputation IP
4,2404000,ET CNC Shadowserver Reported CnC Server IP
5,2404029,ET CNC Shadowserver Reported CnC Server
6,2404300,ET CNC Feodo Tracker Reported CnC Server
7,2500000,ET COMPROMISED Known Compromised or Hostile Host Traffic
8,2520000,ET TOR Known Tor Exit Node Traffic
9,2522000,ET TOR Known Tor Relay/Router (Not Exit) Node Traffic
10,2525000,ET 3CORESec Poor Reputation IP
30,TESTBadIP,TEST Known Bad IP Reputation
31,Apache,Known Apache Bad IP
32,Bots,Known Bots Bad IP
33,CiBadguys,Known Ci-Badguys Bad IP
34,FTP,Known FTP Bad IP
35,IMAP,Known IMAP Bad IP
36,Mail,Known Mail Bad IP
37,SIP,Known SIP Bad IP
38,SSH,Known SSH Bad IP

These are the rules matching custom categories above: /etc/suricata/rules/scirius.rules

drop ip any any -> any any (msg:"TSS Bad Reputation IP"; iprep:any,TESTBadIP,>,99; sid:1; rev:1;)
drop ip any any -> any any (msg:"TSS Known Apache Bad IP"; iprep:any,Apache,>,99; sid:2; rev:1;)
drop ip any any -> any any (msg:"TSS Known Bots Bad IP"; iprep:any,Bots,>,99; sid:3; rev:1;)
drop ip any any -> any any (msg:"TSS Known Ci-Badguys Bad IP"; iprep:any,CiBadguys,>,99; sid:4; rev:1;)
drop ip any any -> any any (msg:"TSS Known FTP Bad IP"; iprep:any,FTP,>,99; sid:5; rev:1;)
drop ip any any -> any any (msg:"TSS Known IMAP Bad IP"; iprep:any,IMAP,>,99; sid:6; rev:1;)
drop ip any any -> any any (msg:"TSS Known Mail Bad IP"; iprep:any,Mail,>,99; sid:7; rev:1;)
drop ip any any -> any any (msg:"TSS Known SIP Bad IP"; iprep:any,SIP,>,99; sid:8; rev:1;)
drop ip any any -> any any (msg:"TSS Known SSH Bad IP"; iprep:any,SSH,>,99; sid:9; rev:1;)

These are the iprep files for each category

# Suricata configuration file SELKS addition.
# This file is added to /etc/suricata/suricata.yaml and overrides
# specific settings

# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules/iprep/
reputation-files:
 - scirius-iprep.list
 - test-iprep.list
 - apache-iprep.list
 - bots-iprep.list
 - cibadguys-iprep.list
 - ftp-iprep.list
 - imap-iprep.list
 - mail-iprep.list
 - sip-iprep.list
 - ssh-iprep.list

Suricata logs confirms the iprep files and rules were successfully loaded:

[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/scirius-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/test-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/apache-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/bots-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/cibadguys-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/ftp-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/imap-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/mail-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/sip-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (reputation.c:635) <Info> (SRepInit) -- Loading reputation file: /etc/suricata/rules/iprep/ssh-iprep.list
[7066] 28/1/2021 -- 16:04:23 - (detect-engine-loader.c:355) <Info> (SigLoadSignatures) -- 1 rule files processed. 9 rules successfully loaded, 0 rules failed
[7066] 28/1/2021 -- 16:04:23 - (util-threshold-config.c:1091) <Info> (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found
[7066] 28/1/2021 -- 16:04:23 - (detect-engine-build.c:1420) <Info> (SigAddressPrepareStage1) -- 9 signatures processed. 9 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only

Few samples from the iprep files:

1.22.8.188,31,100
103.100.175.150,31,100
130.105.248.28,31,100
129.213.165.255,33,100
129.213.203.59,33,100
129.226.169.113,33,100

IPs matching cat 30 are being blocked as expected, but the others are not. Any ideas about what could I doing wrong? I noticed Scirius has one single iprep file, mixing all categories, while I have an iprep file per each category. Is there a limit for how many iprep files Suricata can handle? Should I mix all iprep files into one?

I hope you can help me!

Thank you

pevma commented 3 years ago

Just as a test , can you try putting it all in one rep file ?

ManuelFFF commented 3 years ago

Hi,

Thank you for answering. I was thinking the same thing yesterday, so this morning I have been trying this new config.:

# Suricata configuration file SELKS addition.
# This file is added to /etc/suricata/suricata.yaml and overrides
# specific settings

# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules/iprep/
reputation-files:
 - scirius-iprep.list
 - test-iprep.list

Results:

149.20.4.15,30,100
128.31.0.62,30,100
176.221.42.32,31,100
113.212.69.128,31,100
108.62.59.27,31,100
95.141.17.244,31,100
216.151.137.155,36,100
173.234.225.161,36,100
108.62.56.222,36,100
95.141.17.10,36,100

This leads me to believe that Suricata is not fully reading the iprep files, that there is a limit to the number of lines it can read / load.

Is there anything we can do to fix this issue? Is there a variable or config to modify/remove this limit?

Thank you

pevma commented 3 years ago

I suggest you open an issue - https://forum.suricata.io/t/suricata-and-ip-blacklist/972/18?u=pevma

ManuelFFF commented 3 years ago

Oh, that's a pity. I was hopping that it was something I missed or configured incorrectly. Could you confirm/reproduce the same issue on your end?

Is this the site to report a bug? https://redmine.openinfosecfoundation.org/projects/suricata/issues?set_filter=1&tracker_id=1

ManuelFFF commented 3 years ago

I will submit the bug report. In the meantime, can you please tell me more about using a dataset as suggested by Eric Leblond in here? Will this do the same work as IPREP?

Thanks

ManuelFFF commented 3 years ago

Report submitted https://redmine.openinfosecfoundation.org/issues/4280

If you can, please tell me if datasets will serve the same purpose, if this could avoid the problem I have right now, and if you could give me more details about its implementation.

Thank you

ManuelFFF commented 3 years ago

Hi,

I have a question. The SELKS reputation file /etc/suricata/rules/iprep/scirius-iprep.list, is overwritten on every Scirius restart or any other particular event? You are aware of the issues I'm experiencing with Suricata. I have noticed that scirius-iprep.list has around 21600 IPs/networks to be blocked and it seems to be working fine, so I have been considering to add my custom IPs into that file to see if that way the IPREP mechanism works as I need.

But I also noticed the following. If you check the config below. I changed the default-reputation-path: to /etc/suricata/rules/iprep/ instead of the default previous value /etc/suricata/rules/. After that I moved all iprep lists to the new folder. Well, from time to time, the file scirius-iprep.list it's being created in the old location. 

# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules/iprep/
reputation-files:
 - scirius-iprep.list
 - test-iprep.list

If there is a parallel mechanism checking/overwriting the file scirius-iprep.list, then I might loose my custom IPs when the file it's re-created, plus I am not sure which file will use Suricata: /etc/suricata/rules/scirius-iprep.list or /etc/suricata/rules/iprep/scirius-iprep.list.

Any ideas?

Thank you

ManuelFFF commented 3 years ago

Some updates:

Take for instance IP 95.141.17.10 . If I add that IP to either file /etc/suricata/rules/scirius-iprep.list or /etc/suricata/rules/iprep/scirius-iprep.list , as line # 21605 ( 95.141.17.10,31,100 ), then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.

Other combination is mixing some blacklists into file /etc/suricata/rules/iprep/test-iprep.list. Here IP 95.141.17.10 will be in line # 3136 ( 95.141.17.10,31,100 ). Then restart Suricata and test access from TEST PC behind Suricata, the IP it is NOT being blocked.

However, if I add same IP to file /etc/suricata/rules/iprep/test-iprep.list , as line # 12 ( 95.141.17.10,31,100 ), then restart Suricata and test access from TEST PC behind Suricata, the IP is successfully blocked.

I’m still modifying variables under Host Table to see if with higher values would help IPREP mechanism to work properly.

Current values are:

# Host table:
#
# Host table is used by the tagging and per host thresholding subsystems.
#
host:
  hash-size: 4194304
  prealloc: 1024000
  memcap: 16384mb

Do you advise to keep increasing (doubling) all values or should I stop and try something else? Variable memcap just reached 16GB, but the issue has not been resolved yet, nor I can notice any other impact in Suricata.

I'm currently editing the variables above in /etc/suricata/suricata.yaml. Are these variables being edited/overwritten somewhere else within SELKS? Or do I need to enable something else so Suricata start using values from Host table, because I am not able to see any change so far, after have been changing those values for a while now.

Thank you

ManuelFFF commented 3 years ago

From system logs I see the following:

Feb 01 11:10:56 server1 suricata[2039]: Starting suricata in IDS (af-packet) mode... done.

Can I be sure Suricata is running in IPS mode even with the record above or there is anything I need to check?

Thank you

ManuelFFF commented 3 years ago

Another update

Currently using only one categories file and two reputation files

# IP Reputation
reputation-categories-file: /etc/suricata/rules/scirius-categories.txt
default-reputation-path: /etc/suricata/rules/iprep/
reputation-files:
 - scirius-iprep.list
 - test-iprep.list

I mixed all blacklists into test-iprep.list, having around 50500 IPs in the list. List includes the IPs I used as small sample in previous post. I am including their line numbers

10067   176.221.42.32,31,100
6326    113.212.69.128,31,100
4177    108.62.59.27,31,100
3370    95.141.17.244,31,100
33835   216.151.137.155,36,100
32561   173.234.225.161,36,100
30574   108.62.56.222,36,100
30106   95.141.17.10,36,100

None of these IPs are being blocked by Suricata. Same result if I leave all blacklists separated in individual iprep lists.

However if I move these IPS next to the top in the same list, then restart Suricata, the same IPs are being blocked as expected. Again, I am not an expert, but it looks to me that Suricata is not reading/loading/processing the entire list.

After few tests I have narrowed the working range up to 1170 lines of IPs within the list. If IP falls beyond that point, it won’t be blocked. That’s not a large number of IPs to check.

Unfortunately solution suggested by Victor Julien did not work.

In this article, somebody was testing iprep with data sets up to million entries with positive results! Having a small list with only 1170 entries vs a million entries working fine, makes me think that perhaps Suricata is not reading values properly from the Host table or Host table mechanism is not working properly. Is there a way to check the host table or a command to force loading data (other that USR2 or restart Suricata)?

Any other ideas or anything else I could check or try?

Thanks

ManuelFFF commented 3 years ago

I have been researching on my own about datasets in Suricata. It looks like this works similar to iprepengine, but I'm not sure of quite understand the documentation and I am having a hard time finding examples that I can compare and use as reference.

Could you please be so kind to provide an example of a basic datasetconfig that I can use to replace iprepengine. This does not means that I have given up with IPREP, it's just that I want to try something else in the meantime. I really need to have a mechanism where Suricata can block IPs from one or more blacklists ASAP. Hopefully you guys will be able to find out what is wrong with IPREPin my Suricata.

Probably I will be setting a dataset config in /etc/suricata/selks6-addin.yaml, also creating a dataset or a datarep file and using rules like this to try out DATASET:

From original doc ...

alert dns any any -> any any (dns.query; datarep:dns_string, >, 200, load dns_string.rep, type string; sid:3;)

Customizing ...

drop ip any any -> any any (ip; datarep:ipbl_string, >, 100, load ipbl_string, type string; sid:10;)

I'm not sure what other values I can use for the field where dns.query is, so I am just guessing ip. Not sure if this make sense.

As far as I have seen, dataset seems to be easier than datarep. But I have many questions.

drop ip any any -> any any (msg:"TEST Bad IPs"; dataset:isset, BadIPs; sid:10; rev:1;)

Thank you in advance