search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
199 stars 429 forks source link

Discuss local agent processing #2305

Open gbanasiak opened 2 years ago

gbanasiak commented 2 years ago

Today integrations perform data parsing through ES ingest pipelines with optional local processors added through "advanced options" in Integrations UI. In many deployments this makes perfect sense as we want to keep agent lightweight to not overload the agent host. There are cases however, where agent is deployed on a machine with a lot of resources where we could benefit from local processing to decrease the load of ES ingest nodes.

P1llus commented 2 years ago

I think the best course of action here is to build in support for the ingest pipeline processors in beats if it was something we would want to do, so that it can reuse the exact same formatted ingest pipeline syntax, meaning there is still only 1 copy of the ingestion we need to maintain. That way you can add a button in the Integration UI for "Local Processing" or something similar.

I think @andrewkroh has been playing with something similar?

zez3 commented 2 years ago

@gbanasiak thanks for opening this. @P1llus That one "Local Processing" button option would be ideal.

zez3 commented 2 years ago

@jamiehynds Would you please add this to your roadmap discussion.

jamiehynds commented 2 years ago

Hey @zez3 - will certainly keep this in mind as we iterate on our roadmap. We understand that ingest pipelines may not suit all users (for a variety of reasons) and providing flexibility as to where processing takes place is something we'd like to support. Will keep you updated as we continue to discuss.

theJaspher commented 2 years ago

@jamiehynds what about at least processing things like json, csv, kv, dissect/grok on the agent first, before shipping to the ingest pipeline? This way at least fields are defined on the agent and users can still utilize the Processor section of the integration in Fleet. Things like geo lookup, renaming fields to ECS, scripts, etc can still be done on the ingest pipeline.

Related: #2532

zez3 commented 2 years ago

what about at least processing things like json, csv, kv, dissect/grok on the agent first, before shipping to the ingest pipeline?

What about them? We already have this option in fleet integration policy advanced peocessors.even if a bit cumbersome to squeeze all your parsing in a small edit window(perhaps it should be resizeable? ER?) I use this to manually parse on a few of my biffier agents(running on 128core machine) I think the discussion here is: how can we make use of the integrations locally on the agent? There is no grok available locally, dissect and dns is what I use at this moment

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

zez3 commented 1 year ago

Still relevant

botelastic[bot] commented 6 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

zez3 commented 6 months ago

Go for unify of elasticsearch and beats processors