search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
86 stars 4.92k forks source link

Low throughput rates for Cisco ASA module's Ingest Node pipeline #25768

Open andrewkroh opened 3 years ago

andrewkroh commented 3 years ago

There have been multiple reports of low event rates when using the Cisco ASA Filebeat module. In two independent analyses came to the conclusion was that one processor in the pipeline was taking a significant amount of time compared to the others. This was the date processor with timezone option.

https://github.com/elastic/beats/blob/1da173a9e716715a7a54bb3ff4db05b5c24fc8ce/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml#L105-L127

_nodes/stats metrics in one case showed that about a quarter of the processing time was spent in this processor (analyzed with https://github.com/andrewkroh/go-ingest-node-metrics). We need to investigate why this one processor takes more time and see if we can improve the overall throughput.

109562145-04e4d000-7aac-11eb-98e0-2b0f76629d36

For confirmed bugs, please report:

elasticmachine commented 3 years ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh commented 3 years ago

I think we need to setup a benchmark to replicate the issue. Then test a few changes and measure the results.

One change I'd like to see tested is what happens if the number of date patterns is reduced. And other whether including the timezone option affects the date processor's execution time.

leehinman commented 3 years ago

relates elastic/elasticsearch#73918

botelastic[bot] commented 2 years ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

andrewkroh commented 1 year ago

This date processor change probably affected performance under ES 7.17 and 8.x. We should retest.

https://github.com/elastic/elasticsearch/pull/83764

joegallo commented 1 year ago

I'd expect https://github.com/elastic/elasticsearch/pull/92880 to make a difference here, too.

elasticmachine commented 9 months ago

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)