Data loss due to packet receive errors (MemErrors)

SC4S version: 3.31.0
Host OS: Ubuntu 22.04.4 LTS (Jammy Jellyfish)
Runtime: Docker + systemd

Hello,

I've been having issues related to dropped packets at a buffer level (fortinet_fortios over UDP). Only some data is lost (around 10% loss) and there are no dropped events by sc4s nor custom filters or parsers in place.

After increasing the default parameters like so (following this issue) to be safe:

net.core.rmem_default = 536870912
net.core.rmem_max = 536870912
SC4S_SOURCE_UDP_SO_RCVBUFF=536870912

I've managed to suppress all buffer errors completely. However, I'm still missing events at a rate of aprox. 1000 events per minute (compared to a setup for the same raw data deployed as HF+syslog-ng). Netstat shows the next output:

root@myserver:~# netstat -anus
[...]
Udp:
    6834146 packets received
    287775 packets to unknown port received
    1945378 packet receive errors
    1305 packets sent
    0 receive buffer errors
    0 send buffer errors
    MemErrors: 1945378
[...]

Be aware that the number of receive errors correspond exactly to the number of MemErrors. I'm wondering how can I fix this issue. I've also tried raising the number of workers from 4 to 8, unsuccessfully.

Thank you for your support.

splunk / splunk-connect-for-syslog

Data loss due to packet receive errors (MemErrors) #2604