TCP reassembly gaps

[Noname2244]

Could this be a problem?

tap interface is a intel NIC with optimizations enabled all offloading disabled etc. General low cpu and low memory usage.

[pevma]

What do you mean “optimization enabled “?

There seems to be loads of invalids ... can you do a tcpdump and see if something wired is up ?

Is that on SELKS 4 or 5 ?

-- Regards, Peter Manev

On 2 Nov 2018, at 16:30, Noname2244 notifications@github.com wrote:

Could this be a problem?

tap interface is a intel NIC with optimizations enabled all offloading disabled etc. General low cpu and low memory usage.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Noname2244]

I ran the tuning scripts supplied with SELKS.

Im trying out SELKS5 right now, but had the same "issue" on SELKS 4.

could it be the Intel NIC drivers?

[Noname2244]

Its still increasing linearly.

[pevma]

Not sure if it is driver related. Did you manage to have a look with a pcap/tcpdump like previously suggested?

[Noname2244]

well what should I look for in a dump ? since its that tap interface there is lots of traffic incoming, well not that much but enough to need a filter with tcpdump.

[Noname2244]

Okay we have some dropped packets

sudo netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
ens3f0    1500 606201007      0  24738 0           129      0      0      0 BMRU
ens3f1    1500   625877      0   2924 0        339409      0      0      0 BMRU
lo       65536 29771014      0      0 0      29771014      0      0      0 LRU

[pevma]

There seems to be a big number of decoder invalid packets - if you can do a tcpdump and look into it to see if you see anything odd (example wireshark will complain on a lot of things). Also check if have vlan tags in the traffic - you could also fetch the last update in /var/log/suricata/stats.log and share that here so we can have a look ?

[Noname2244]

I will get a tcpdump.

suricata.log

`------------------------------------------------------------------------------------
Date: 11/6/2018 -- 13:51:05 (uptime: 4d, 21h 44m 44s)
------------------------------------------------------------------------------------
Counter                                    | TM Name                   | Value
------------------------------------------------------------------------------------
capture.kernel_packets                     | Total                     | 630454237
capture.kernel_drops                       | Total                     | 20619657
decoder.pkts                               | Total                     | 609834636
decoder.bytes                              | Total                     | 131947407836
decoder.invalid                            | Total                     | 209657
decoder.ipv4                               | Total                     | 609197319
decoder.ipv6                               | Total                     | 46181
decoder.ethernet                           | Total                     | 813429454
decoder.tcp                                | Total                     | 358679362
decoder.udp                                | Total                     | 249003830
decoder.icmpv4                             | Total                     | 1555576
decoder.icmpv6                             | Total                     | 4072
decoder.vlan                               | Total                     | 609834511
decoder.teredo                             | Total                     | 9
decoder.avg_pkt_size                       | Total                     | 216
decoder.max_pkt_size                       | Total                     | 3017
flow.tcp                                   | Total                     | 4527242
flow.udp                                   | Total                     | 11409383
flow.icmpv4                                | Total                     | 217741
flow.icmpv6                                | Total                     | 1264
decoder.ipv4.opt_pad_required              | Total                     | 651
decoder.ipv6.zero_len_padn                 | Total                     | 3451
decoder.vlan.unknown_type                  | Total                     | 209657
tcp.sessions                               | Total                     | 1301815
tcp.syn                                    | Total                     | 1427737
tcp.synack                                 | Total                     | 3128203
tcp.rst                                    | Total                     | 2649481
tcp.pkt_on_wrong_thread                    | Total                     | 99593304
tcp.stream_depth_reached                   | Total                     | 2
tcp.reassembly_gap                         | Total                     | 79866
tcp.overlap                                | Total                     | 34012608
detect.alert                               | Total                     | 1455
app_layer.flow.http                        | Total                     | 115440
app_layer.tx.http                          | Total                     | 9039816
app_layer.flow.tls                         | Total                     | 25503
app_layer.flow.ssh                         | Total                     | 1
app_layer.flow.smb                         | Total                     | 12
app_layer.tx.smb                           | Total                     | 537
app_layer.flow.dns_tcp                     | Total                     | 4
app_layer.tx.dns_tcp                       | Total                     | 8
app_layer.flow.ntp                         | Total                     | 120291
app_layer.flow.krb5_tcp                    | Total                     | 510
app_layer.tx.krb5_tcp                      | Total                     | 510
app_layer.flow.failed_tcp                  | Total                     | 312
app_layer.flow.dcerpc_udp                  | Total                     | 9
app_layer.flow.dns_udp                     | Total                     | 325273
app_layer.tx.dns_udp                       | Total                     | 447945
app_layer.flow.enip_udp                    | Total                     | 429
app_layer.tx.enip_udp                      | Total                     | 1194
app_layer.tx.ntp                           | Total                     | 28975
app_layer.flow.krb5_udp                    | Total                     | 1752
app_layer.tx.krb5_udp                      | Total                     | 555
app_layer.flow.failed_udp                  | Total                     | 10961629
flow_mgr.closed_pruned                     | Total                     | 124213
flow_mgr.new_pruned                        | Total                     | 15902850
flow_mgr.est_pruned                        | Total                     | 126136
flow.spare                                 | Total                     | 10037
flow.tcp_reuse                             | Total                     | 70
flow_mgr.flows_checked                     | Total                     | 102
flow_mgr.flows_notimeout                   | Total                     | 102
flow_mgr.rows_checked                      | Total                     | 65536
flow_mgr.rows_skipped                      | Total                     | 65351
flow_mgr.rows_empty                        | Total                     | 86
flow_mgr.rows_maxlen                       | Total                     | 2
tcp.memuse                                 | Total                     | 9175040
tcp.reassembly_memuse                      | Total                     | 4373644
http.memuse                                | Total                     | 117843
flow.memuse                                | Total                     | 7987920

`

[pevma]

actually -

decoder.invalid                            | Total                     | 209657
....
decoder.vlan.unknown_type                  | Total                     | 209657

the total number of invalids is not that high compared to the total number seen. It also seems to be related to some specific VLAN ?

[Noname2244]

Im spanning a vlan , so we have tagged frames, is that a problem ?

[pevma]

No that is not a problem. The problem seems to be with a specific tag or type of vlan - not all - from what i see from the logs.

Also - can you increase the ringsize buffers on the interface config in /etc/suricata/selks5-interfaces-config.yaml then restart and have look if things are better after a bit

[Noname2244]

Changing af-packet ringsize to 8192 fixed it!

[Gatall]

I have a similar problem Sometimes may appear "Decoder invalid" about 2-4 max. Traffic sniffing by mikrotik. Start suricata by script Value ring-size dont fix it to me. Thx!

[pevma]

It could be valid traffic missing for example. Do you have any other drops in /var/log/suricata/stats.log (latest section update)?

[Gatall]

decoder.invalid 194 now :( I also tried to send traffic so: / usr/bin/tzsp2pcap -f -s 3000 | /usr/bin/tcpreplay --topspeed -i eth10 - ETH10 is dimmy enterface with mtu 3000 and i have like this

[pevma]

I dont think in that case 194 is something to worry about - 0.0007% of 27266624. finding out exactly what triggers it would be nice - but very hard without a pcap to reproduce it? You have 194 IPv4 truncated pakets decoder.ipv4.trunc_packet

[Gatall]

I have achieved the following. I changed the speed of tzsp2pcap, mtu & port. Should I be worried or it is normal for Suricata ?

[pevma]

Seems ok to me - no drops and big memcap hits ...