Can not alert DNS rules or other application layer protocols

[Leonardo-DiCaprio]

Reference from #26

I changed my maxium number of rules to 10000, and nothing else in suricata.yaml has been changed.Then I link my NIC eno1( which using kernel driver) to 0000:03:00.0(which using igb_uio module) with a Fiber. And try to send traffic through tcpreplay by: tcpreplay -i eno1 mytest.pcap (I got this pcap file through wireshark. It includes enough DNS record) It seems suricata start to work. I am wondering how DPDK-suricata detect DNS or HTTP traffic as DPDK works in the second and third layer. I enable dns-log in suricata.yaml but I didn't get any record in dns.log. What should I do if I want to detect DNS traffic?

and

Actually, the dpdk-suricata shows that I receive most of the traffic(not all of them, some traffic are too long to send through tcpreplay) :

6/8/2020 -- 12:58:35 - <Notice> - DPDK Started in IDS Mode!!!
^C6/8/2020 -- 13:02:08 - <Notice> - Signal Received.  Stopping engine.
6/8/2020 -- 13:02:08 - <Notice> -  --- thread stats for Intf: 0 to 0 --- 
6/8/2020 -- 13:02:08 - <Notice> -  +++ ACL +++
6/8/2020 -- 13:02:08 - <Notice> -  - non IP 1187
6/8/2020 -- 13:02:08 - <Notice> -  +++ ipv4 7169 +++
6/8/2020 -- 13:02:08 - <Notice> -  - lookup: success 7169, fail 0
6/8/2020 -- 13:02:08 - <Notice> -  - result: hit 6068, miss 1101
6/8/2020 -- 13:02:08 - <Notice> -  +++ ipv6 325 +++
6/8/2020 -- 13:02:08 - <Notice> -  - lookup: success 325, fail 0
6/8/2020 -- 13:02:08 - <Notice> -  - result: hit 4, miss 321
6/8/2020 -- 13:02:08 - <Notice> -  +++ ring +++
6/8/2020 -- 13:02:08 - <Notice> -  ERR: full 0, enq 0, tx 0
6/8/2020 -- 13:02:08 - <Notice> -  +++ port 0 +++
6/8/2020 -- 13:02:08 - <Notice> -  - index 0 pkts RX **8681** TX 0 MISS 0
6/8/2020 -- 13:02:08 - <Notice> -  - Errors RX: 0 TX: 0 Mbuff: 0
6/8/2020 -- 13:02:08 - <Notice> -  - Queue Dropped pkts: 0
6/8/2020 -- 13:02:08 - <Notice> - ----------------------------------
6/8/2020 -- 13:02:08 - <Notice> - Stats for '0000:03:00.0':  pkts: 0, drop: 0 (-nan%), invalid chksum: 0

But nothing has been detected yet. And I get a empty dns.log and a eve.log, which shows that only empty packet was received. And I figure out that even though I dont push any traffic to DPDK interface, the dpdk-suricata will receive those empty packet also (maybe it is a tiny bug of my fiber topology ). So I think I dont get any event analyzed actually.

Now I know the original Suricata could work out these. So I think there is something wrong with my configuration(suricata.yaml and dpdk-suricata.ini, which I uploaded above), or my building of dpdk-suricata.

vipinpv85:

You are still not sharing the following information

whether there is DNS rule actually added to the system or not. whether there is DNS request and reply send from pcap or not.

Incorrect understanding. It is not empty packets, but there are no events or logs to report. Check if DNS rule is applied or not. yes, please check if you have added DNS rules in the logs.

I am pretty sure DNS rules are actually added to the system, for: in suricata.yaml:

rule-files:
 - dns-events.rules     # available in suricata sources under rules dir
 - test.rules

and in dns-events.rules:

# Request Flood Detected
# alert dns any any -> any any (msg:"SURICATA DNS request flood detected"; flow:to_server; app-layer-event:dns.flooded; sid:2240007; rev:1;)
# Per-flow (state) memcap reached. Relates to the app-layer.protocols.dns.state-memcap setting.
# alert dns any any -> any any (msg:"SURICATA DNS flow memcap reached"; flow:to_server; app-layer-event:dns.state_memcap_reached; sid:2240008; rev:2;)

alert dns any any -> any any (msg:"hit DNS";sid:2240008; rev:2;)

and in test.rules: alert http any any -> any any (msg:"hit dns google"; content:"g";sid:2100498; rev:7;) What's more: I checked whether DNS request and reply have been sent to this NIC interface by using tcpdump(in kernel mode) and pdump(in dpdk mode). And answer is yes.

I check the yaml config, but all things seem right for me.

So I dont know what's wrong with my config. And there it is. suricata.yaml.txt

[Leonardo-DiCaprio]

And build-info seems incorrectly:

This is Suricata version 3.0 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 DPDK_INTEL AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON TLS 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 5.4.0 20160609, C version 201112
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.18, linked against LibHTP v0.5.18

Suricata Configuration:
  AF_PACKET support:                       no
  DPDK_INTEL support:                      yes
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no

  Suricatasc install:                      yes

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /usr/local/var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /usr/local/var

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native -DHAVE_DPDKINTEL -I/home/adm1n/dpdk/x86_64-native-linuxapp-gcc/include -include /home/adm1n/dpdk/x86_64-native-linuxapp-gcc/include/rte_config.h 
  PCAP_CFLAGS                               -I/usr/include

I compare this with original build info of suricata, It seems that AF_PACKET support and NFQueue support should be yes. But I got a no there. Will this influence DPDK_INTEL module work?

[vipinpv85]

Reference from #26

vipinpv85:

You are still not sharing the following information whether there is DNS rule actually added to the system or not. whether there is DNS request and reply send from pcap or not.

Share the log, showcasing the DNS is set for suricata. For the set rule, PCAP is replayed (ie DNS packets)

alert dns any any -> any any (msg:"hit DNS";sid:2240008; rev:2;)

[vipinpv85]

And build-info seems incorrectly: what is incorrect in the build-info?

[vipinpv85]

I have tested the feature internally and found the following

• https://suricata-ids.org/tag/dns/ (issues with DNS parsing in suricata is only fixed in 4.1.2)
• manually add rules and cross check if the DPDK ACL - passed
• manually create DNS packets and send them - passed
• dns rules are hit - fail
• is there dns.log - no
• is there event.log generated - yes

reason:

1. the packet is identified as UDP when bypassing DPDK ACL filter file: /usr/local/var/log/suricata/eve.json

   {"timestamp":"2020-08-06T15:14:28.229426+0530","flow_id":93828676553296,"event_type":"dns","src_ip":"192.168.1.1","src_port":53,"dest_ip":"192    .168.1.101","dest_port":57311,"proto":"UDP","dns":{"type":"answer","id":9074,"rcode":"NOERROR","rrname":"www3.l.google.com","rrtype":"A","ttl"    :262,"rdata":"173.194.35.41"}}

2. Suricata is treating rules as TCP

6/8/2020 -- 15:30:24 - - - Proto 0x6 Mask 0xFF 6/8/2020 -- 15:30:24 - - - SRC IP 0 Mask ffffffff 6/8/2020 -- 15:30:24 - - - DST IP 0 Mask ffffffff 6/8/2020 -- 15:30:24 - - - Proto 0x6 Mask 0xFF 6/8/2020 -- 15:30:24 - - - SRC IP 0 Mask ffffffff 6/8/2020 -- 15:30:24 - - - DST IP 0 Mask ffffffff 6/8/2020 -- 15:30:24 - - - Proto 0x6 Mask 0xFF 6/8/2020 -- 15:30:24 - - - SRC IP 0 Mask ffffffff 6/8/2020 -- 15:30:24 - - - DST IP 0 Mask ffffffff

patch to test:

# git diff src/source-dpdkintel.c
diff --git a/suricata-3.0/src/source-dpdkintel.c b/suricata-3.0/src/source-dpdkintel.c
index 6793152..b40581f 100644
--- a/suricata-3.0/src/source-dpdkintel.c
+++ b/suricata-3.0/src/source-dpdkintel.c
@@ -166,7 +166,7 @@ FilterPackets(struct rte_mbuf *m, uint32_t *res, uint16_t inPort)
                 &data, res, 1, 1, RTE_ACL_CLASSIFY_SSE) == 0)) {
                 dpdkStats [inPort].ipv4_pkt_success++;
                 (*res == 0) ? dpdkStats [inPort].ipv4_pkt_aclmiss++ : dpdkStats [inPort].ipv4_pkt_aclhit++;
-
+               *res = 1;
                 SCLogDebug(" post ipv4 acl result %x", *res);
             } else {
                 dpdkStats [inPort].ipv4_pkt_fail++;
@@ -181,6 +181,7 @@ FilterPackets(struct rte_mbuf *m, uint32_t *res, uint16_t inPort)
                 &data, res, 1, 1, RTE_ACL_CLASSIFY_SSE) == 0)) {
                 dpdkStats [inPort].ipv6_pkt_success++;
                 (*res == 0) ? dpdkStats [inPort].ipv6_pkt_aclmiss++ : dpdkStats [inPort].ipv6_pkt_aclhit++;
+               *res = 1;

                 SCLogDebug(" ipv6 acl result %x", *res);
             } else {

Logs:

6/8/2020 -- 15:02:08 - - DPDK Version: DPDK 18.11.7 6/8/2020 -- 15:02:08 - - ----- Global DPDK-INTEL Config ----- 6/8/2020 -- 15:02:08 - - Number Of Ports : 2 6/8/2020 -- 15:02:08 - - Operation Mode : IPS 6/8/2020 -- 15:02:08 - - Port:0, Map:1 6/8/2020 -- 15:02:08 - - Port:1, Map:0 6/8/2020 -- 15:02:08 - - ------------------------------------ 6/8/2020 -- 15:02:10 - - ----- Match Pattern ---- 6/8/2020 -- 15:02:10 - - http: 0 6/8/2020 -- 15:02:10 - - ftp: 0 6/8/2020 -- 15:02:10 - - * tls: 0 *6/8/2020 -- 15:02:10 - - dns: 1* 6/8/2020 -- 15:02:10 - - smtp: 0 6/8/2020 -- 15:02:10 - - ssh: 0 6/8/2020 -- 15:02:10 - - smb: 0 6/8/2020 -- 15:02:10 - - smb2: 0 6/8/2020 -- 15:02:10 - - dcerpc:0 6/8/2020 -- 15:02:10 - - tcp: 1 6/8/2020 -- 15:02:10 - - udp: 1 6/8/2020 -- 15:02:10 - - sctp: 0 6/8/2020 -- 15:02:10 - - icmpv4:0 6/8/2020 -- 15:02:10 - - icmpv6:0 6/8/2020 -- 15:02:10 - - gre: 0 6/8/2020 -- 15:02:10 - - raw: 0 6/8/2020 -- 15:02:10 - - ipv4: 0 6/8/2020 -- 15:02:10 - - * ipv6: 0 6/8/2020 -- 15:02:10 - - -----------------------

6/8/2020 -- 15:02:53 - - --- thread stats for Intf: 1 to 0 --- 6/8/2020 -- 15:02:53 - - +++ ACL +++ 6/8/2020 -- 15:02:53 - - - non IP 0 6/8/2020 -- 15:02:53 - - +++ ipv4 8 +++ 6/8/2020 -- 15:02:53 - - - lookup: success 8, fail 0 6/8/2020 -- 15:02:53 - - - result: hit 0, miss 8 6/8/2020 -- 15:02:53 - - +++ ipv6 0 +++ 6/8/2020 -- 15:02:53 - - - lookup: success 0, fail 0 6/8/2020 -- 15:02:53 - - - result: hit 0, miss 0 6/8/2020 -- 15:02:53 - - +++ ring +++ 6/8/2020 -- 15:02:53 - - ERR: full 0, enq 0, tx 0 6/8/2020 -- 15:02:53 - - +++ port 1 +++ 6/8/2020 -- 15:02:53 - - - index 1 pkts RX 0 TX 8 MISS 0 6/8/2020 -- 15:02:53 - - - Errors RX: 0 TX: 0 Mbuff: 0 6/8/2020 -- 15:02:53 - - - Queue Dropped pkts: 0 6/8/2020 -- 15:02:53 - - ----------------------------------

[vipinpv85]

attaching the pcap file used in my internal test.

dns.pcapng.txt

[vipinpv85]

I have asked multiple times in https://github.com/vipinpv85/DPDK-Suricata_3.0/issues/26#issuecomment-669711303 to check if the environment is proper and build Suricata without ./configure --enable-dpdkintel but with ./configure. So that it can be tested for presence of errors or dns.log. I have not found any logs, answers or meeting invite from you.

Note: without having you been online via skype or zoom or pastebin, I find it very difficult to debug for you.

[vipinpv85]

@Leonardo-DiCaprio I have added a fix to overcome the limitation from Suricata 3.0. The commit details is in https://github.com/vipinpv85/DPDK-Suricata_3.0/commit/b7c555bcfb8ab707b1902c911b540819cfd48edc . Hope this will help you rather than trying to fix for Suricata 3.0.

[vipinpv85]

Bug in Suricata 3.0 in aprsing and setting up flow rules for DNS. Fixed in DPDK ACL to force for lookup by https://github.com/vipinpv85/DPDK-Suricata_3.0/commit/b7c555bcfb8ab707b1902c911b540819cfd48edc

[Leonardo-DiCaprio]

This commit worked for me. DPDK-Suricata can filter DNS traffic now. Thanks @vipinpv85 And it seems like that the DNS traffic is still marked as UDP packet in fast.log:

08/07/2020-14:39:35.231043  [**] [1:2100498:7] hit dns google [**] [Classification: (null)] [Priority: 3] {UDP} 10.0.1.1:53 -> 10.1.0.2:53
08/07/2020-14:39:35.231097  [**] [1:2100498:7] hit dns google [**] [Classification: (null)] [Priority: 3] {UDP} 10.0.1.1:53 -> 10.1.0.2:53
08/07/2020-14:39:35.231128  [**] [1:2100498:7] hit dns google [**] [Classification: (null)] [Priority: 3] {UDP} 10.0.1.1:53 -> 10.1.0.2:53

[vipinpv85]

This commit worked for me. DPDK-Suricata can filter DNS traffic now. Thanks @vipinpv85 And it seems like that the DNS traffic is still marked as UDP packet in fast.log:

08/07/2020-14:39:35.231043  [**] [1:2100498:7] hit dns google [**] [Classification: (null)] [Priority: 3] {UDP} 10.0.1.1:53 -> 10.1.0.2:53
08/07/2020-14:39:35.231097  [**] [1:2100498:7] hit dns google [**] [Classification: (null)] [Priority: 3] {UDP} 10.0.1.1:53 -> 10.1.0.2:53
08/07/2020-14:39:35.231128  [**] [1:2100498:7] hit dns google [**] [Classification: (null)] [Priority: 3] {UDP} 10.0.1.1:53 -> 10.1.0.2:53

there is no dns.log and parser in Suricata calls out as UDP and not DNS.

[Leonardo-DiCaprio]

And there are some pieces of dns.log as supplement:

08/07/2020-14:59:53.884763 [**] Query TX 948e [**] dns.google.com [**] AAAA [**] 10.0.0.1:53 -> 10.1.0.2:53
08/07/2020-14:59:53.884888 [**] Query TX 00a7 [**] dns.google.com [**] A [**] 10.0.0.1:53 -> 10.1.0.2:53
08/07/2020-14:59:53.885071 [**] Query TX bafe [**] dns.google [**] AAAA [**] 10.0.0.1:53 -> 10.1.0.2:53
08/07/2020-14:59:53.885410 [**] Query TX f7f7 [**] dns.google [**] A [**] 10.0.0.1:53 -> 10.1.0.2:53

[vipinpv85]

are you saying you got dns.logs by ./configure --enable-dpdkintel?

ls /usr/local/var/log/suricata/ -lrtha
total 84K
drwxr-xr-x 4 root root 4.0K Apr 10  2019 ..
-rw-r--r-- 1 root root    0 Apr 10  2019 http.log
-rw-r--r-- 1 root root 4.1K Aug  6 17:25 fast.log
-rw-r--r-- 1 root root  41K Aug  6 17:25 stats.log
-rw-r--r-- 1 root root  19K Aug  6 17:25 eve.json
drwxr-xr-x 2 root root 4.0K Aug  6 17:26 .

rule

alert dns any any -> any any (msg:"BLACKLISTED DOMAIN"; content:"|7F 00 00 01|"; sid:1;)
alert dns any any -> any any (msg:"hit DNS";sid:2240008; rev:2;)
alert dns $HOME_NET any -> any any (msg:"ET DNS Query for google.com"; dns_query; content:"google.com"; nocase; isdataat:!1,relative; reference:ur
l,google.com; classtype:policy-violation; sid:20000000; rev:1;)

in eve.log

{"timestamp":"2020-08-06T17:25:50.677521+0530","flow_id":93999306247040,"event_type":"alert","src_ip":"192.168.1.101","src_port":55365,"dest_ip":"
192.168.1.1","dest_port":53,"proto":"UDP","alert":{"action":"allowed","gid":1,"signature_id":2240008,"rev":2,"signature":"hit DNS","category":"","
severity":3}}

fast.log

 08/06/2020-17:25:48.681425  [**] [1:2240008:2] hit DNS [**] [Classification: (null)] [Priority: 3] {UDP} 192.168.1.101:55365 -> 192.168.1.1:53

[Leonardo-DiCaprio]

are you saying you got dns.logs by ./configure --enable-dpdkintel?

I got dns.log by pulling latest code https://github.com/vipinpv85/DPDK-Suricata_3.0/commit/b7c555bcfb8ab707b1902c911b540819cfd48ed Obviously @vipinpv85 fix this bug in that commit.

[vipinpv85]

are you saying you got dns.logs by ./configure --enable-dpdkintel?

I got dns.log by pulling latest code b7c555b Obviously @vipinpv85 fix this bug in that commit.

It same code I run too. Do you edit suricata..yanl or have different rules

[Leonardo-DiCaprio]

It same code I run too. Do you edit suricata..yanl or have different rules

Not at all. I just change to the latest code after I see this comment https://github.com/vipinpv85/DPDK-Suricata_3.0/issues/27#issuecomment-669884672 Then I got my dns.log... And I think the test rule I used is rougher than yours. It just alert all dns traffic. Emmm how come can it be the rules' fault. alert dns any any -> any any (msg:"hit dns google"; content:"g";sid:2100498; rev:7;)

I will try to debug this project to find out if there anything wrong with my dpdk or Linux enviorment as you say you run the same code and it worked... If there are any difference, I will pasting there.

[vipinpv85]

It same code I run too. Do you edit suricata..yanl or have different rules

Not at all. I just change to the latest code after I see this comment #27 (comment) Then I got my dns.log... And I think the test rule I used is rougher than yours. It just alert all dns traffic. Emmm how come can it be the rules' fault. alert dns any any -> any any (msg:"hit dns google"; content:"g";sid:2100498; rev:7;)

I will try to debug this project to find out if there anything wrong with my dpdk or Linux enviorment as you say you run the same code and it worked... If there are any difference, I will pasting there.

default config in suricata.yaml is

 242   # a line based log of DNS requests and/or replies (no alerts)
 243   - dns-log:
 244       enabled: no
 245       filename: dns.log
 246       append: yes
 247       #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

this will not generate dns.log. I am surprised without changing this how your system generating the same.

[Leonardo-DiCaprio]

Actually my config is:

 242   # a line based log of DNS requests and/or replies (no alerts)
 243   - dns-log:
 244       enabled: yes
 245       filename: dns.log
 246       append: yes
 247       #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

When I said I didn't change the suricata.yaml, I mean I didn't change it before and after it generates the dns.log. But I enabled the dns.log at the very beginning, as I mentioned in https://github.com/vipinpv85/DPDK-Suricata_3.0/issues/26#issuecomment-668365597

Since this issue is transferred from #26 , I didn't repeat that message. Sry to lead you misunderstand.

[vipinpv85]

@Leonardo-DiCaprio question is clearly asked have you have you changed anything for dns.log. checkign your comment there is no mention of the same.

Sorry to bother.
I changed my maxium number of rules to 10000, and nothing else in suricata.yaml has been changed.Then I link my NIC eno1( which using kernel driver) to 0000:03:00.0(which using igb_uio module) with a Fiber. And try to send traffic through tcpreplay by:

tcpreplay -i eno1 mytest.pcap (I got this pcap file through wireshark. It includes enough DNS record)

It seems suricata start to work.

I am wondering how DPDK-suricata detect DNS or HTTP traffic as DPDK works in the second and third layer. I enable dns-log in suricata.yaml but I didn't get any record in dns.log. What should I do if I want to detect DNS traffic?