high load generated by unifiedbeat

[slayer01]

dear cleesmith,

first of all, thank you very much for your effort.

I installed unifiedbeat on my snort box, my snort version is 2.9.8.0 its running on centos 7. Actually there are two unified log files:

-rw------- 1 snort snort 229K 10. Feb 10:35 snort.log.1455026918
-rw------- 1 snort snort  918 10. Feb 11:01 snort.log.1455097021

an top looks like this:

 PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                        
17321 root      20   0  145952  97140   5016 S 67,2  0,6   1:55.67 unifiedbeat                                                                                                                                    
17319 root      20   0  145952  97140   5016 R 66,2  0,6   1:43.55 unifiedbeat                                                                                                                                    
17317 root      20   0  145952  97140   5016 R 57,6  0,6   1:31.90 unifiedbeat                                                                                                                                    
17318 root      20   0  145952  97140   5016 S 15,6  0,6   1:44.70 unifiedbeat                                                                                                                                    
17320 root      20   0  145952  97140   5016 S  6,6  0,6   1:26.69 unifiedbeat                                                                                                                                    
17326 root      20   0  145952  97140   5016 S  6,6  0,6   1:45.11 unifiedbeat                                                                                                                                    
17332 root      20   0  145952  97140   5016 S  5,6  0,6   1:43.52 unifiedbeat                                                                                                                                    
17325 root      20   0  145952  97140   5016 S  4,6  0,6   1:51.60 unifiedbeat                                                                                                                                    
17328 root      20   0  145952  97140   5016 S  4,3  0,6   1:53.96 unifiedbeat                                                                                                                                    
17322 root      20   0  145952  97140   5016 S  4,0  0,6   1:25.09 unifiedbeat                                                                                                                                    
17323 root      20   0  145952  97140   5016 S  3,6  0,6   1:08.53 unifiedbeat                                                                                                                                    
17017 snort     20   0 1548952 767760   5824 S  2,6  4,7   2:26.04 snort                                                                                                                                          
17327 root      20   0  145952  97140   5016 S  2,6  0,6   1:55.36 unifiedbeat                                                                                                                                    
17331 root      20   0  145952  97140   5016 S  2,0  0,6   1:46.41 unifiedbeat                                                                                                                                    
17314 root      20   0  145952  97140   5016 S  1,3  0,6   1:31.76 unifiedbeat                                                                                                                                    
17315 root      20   0  145952  97140   5016 S  0,7  0,6   0:01.04 unifiedbeat  

so you see up to 15 unifiedbeat threads, first 5-10 use cpu about 20 - 70%. Is that normal behaviour?

Here is my unifiedbeat.yml

unifiedbeat:
  geoip2_path: "var/GeoIP/GeoLite2-City.mmdb"
  rules:
    gen_msg_map_path: "/etc/snort/gen-msg.map"
    paths:
      - "/etc/snort/rules/*.rules"
  sensor:
    -
      paths:
        - /var/log/snort/snort.log*

      fields:
        sensor_hostname: ultrasnort1
        sensor_interface: em2
        sensor_type: snort

      fields_under_root: true

      ignore_older: 24h

      scan_frequency: 20s

output:

  logstash:
    hosts: ["logstash.example.de:5044"]
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash.example.de.crt"]

shipper:
  name: nucy

logging:

  to_files: true

  files:
    path: /var/log/unifiedbeat

    name: unifiedbeat

    keepfiles: 3

  level: info
`

[cleesmith]

Unifiedbeat is based on an older clone of Filebeat. The beats team have made many changes to filebeat since Nov 2015. Some of their changes affect how many files remain open and for how long. Once both libbeat and filebeat settle down, I will apply those changes to unifiedbeat.

Go, in general, and all of the beats do use cpu/memory resources as much as possible ... it's seen as a feature usually. Filebeat is/was aggressive about prospecting/harvesting files ... perhaps some of the config settings can lessen it's behavior. I haven't noticed the high cpu usage to adversely affect the sensor's ability to capture alerts. Have you?

Typically, with snort/suricata once a unified2 log file rotates no new data is ever added, so this is probably a change that is needed for unifiedbeat. Filebeat is focused on syslogs so it's behavior is different. Therefore, I will leave this as an open issue that needs attention. Thanks.

[cleesmith]

I downloaded Filebeat 1.1.0 and there is less cpu usage (almost zero after indexing), but there are still at least 9 threads always running:

top - 02:52:27 up 82 days, 10:46,  3 users,  load average: 0.04, 0.03, 0.05
Threads: 209 total,   1 running, 208 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  16361156 total,  8233220 used,  8127936 free,   220088 buffers
KiB Swap: 16703484 total,        0 used, 16703484 free.  2912640 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                             
27380 elastic+  20   0 7901496 4.434g  24888 S  0.0 28.4   0:00.06 java                                                                                                                                
27381 elastic+  20   0 7901496 4.434g  24888 S  0.0 28.4   0:00.06 java                                                                                                                                
27870 elastic+  20   0 7901496 4.434g  24888 S  0.0 28.4   0:08.37 java                                                                                                                                
28384 elastic+  20   0 7901496 4.434g  24888 S  0.0 28.4   0:00.00 java                                                                                                                                
28415 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.04 filebeat                                                                                                                            
28416 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.04 filebeat                                                                                                                            
28417 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.02 filebeat                                                                                                                            
28418 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.03 filebeat                                                                                                                            
28419 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.00 filebeat                                                                                                                            
28420 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.00 filebeat                                                                                                                            
28421 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.03 filebeat                                                                                                                            
28422 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.02 filebeat                                                                                                                            
28423 root      20   0  286460  11940   4564 S  0.0  0.1   0:00.02 filebeat                                                                                                                            
  509 syslog    20   0  255840   5092   1052 S  0.0  0.0   0:00.02 rsyslogd

So it looks like I need to spend some time applying the most recent changes from filebeat to unifiedbeat. This is a side project so I am not sure when I get to do those changes.

[slayer01]

yes, your top looks much better. Looking forward to update with your changes. thanks

[cleesmith]

Version 2 is no longer a clone of filebeat. Filebeat, while great for syslogs, was not appropriate for spooling/indexing unified2 files. It now uses less resources. Let me know if you notice anything.