Elevated Memory Utilization and Errors in Filebeat When Integrating External MISP CTI Log Source

[sun8q]

Summary: Our Elastic setup incorporates two MISP CTI log sources— one internal and one external—both managed within the Henkel environment. Upon integrating these MISP instances using the filebeat API method, we have identified a notable increase in memory utilization exclusively on the external MISP log source. This surge occurs when the external MISP sends logs to logstash shippers, prompting suspicions of a potential bug in the filebeat component of the external MISP.

Outlined below are the steps we undertook to address and investigate the issue:

Currently, the integration of these two MISP log sources is achieved through filebeat using the API method. A dedicated filebeat server has been established, creating distinct services for the INTERNAL MISP and EXTERNAL MISP. The filebeat service has been enabled, initiating the collection of logs from both MISP instances, which are then ingested into logstash shippers. Post-ingestion, logs from both MISP instances are successfully being ingested into Elastic SIEM. However, only the external MISP filebeat server is experiencing a high memory utilization of approximately 8GB, coupled with errors observed in logstash shippers. In an effort to isolate the issue and confirm its association with the external MISP, we conducted the following test:

a. The internal MISP server was installed on Server A and configured to send logs to logstash shippers LSS1 and LSS2. b. Simultaneously, the external MISP server was installed on Server B, configured to send logs to the same logstash shippers (LSS1 and LSS2). c. Despite these adjustments, the memory utilization on the external MISP server remained high, and errors persisted in logstash shippers.

Filebeat version is 8.11

@elastic-data-integration @bug

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[sun8q]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) Team: Team:<@Elastic-Data-Integration> @elastic-data-integration

[skorpion8909]

I am having the same issue with filebeat running out of memory after 2 minutes from the start(hard cap is set to 6 GB in filebeat.service). It just started randomly today. The last change made was adding additional netflow shippers(An additional 50GB a day)

OS:EuroLinux 8.9 (Monaco) Filebeat: 8.11.3 CPU: 2 MEM:12 GB Modules: Netflow, Fortinet, System, Cisco,

[skorpion8909]

It started to work properly after changing the number of "harvester_limit:" from 100 to 20. scan_frequency: 10s harvester_limit: 20 backoff: 2s max_backoff: 10s backoff_factor: 2

[skorpion8909]

It started occurring again today. I am starting to thing this problem is related to performance issues while reading the syslog files by Fortinet or cisco module.

After disabling module fortinet(it collects fortinet logs from files) Filebeat started working correctly.

Also, i notice that when filebeat(with fortinet module enabled) is rising memeory usage very fast i can see in logs

log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":145},"message":"Connec
tion to backoff(elasticsearch(https://someurl:9200)) established","service.name":"filebeat","ecs.version":"1.6.0"}