search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.62k stars 24.64k forks source link

DNS Processor for Ingest Pipelines #91624

Open MakoWish opened 1 year ago

MakoWish commented 1 year ago

Description

TL:DR - There needs to be a DNS processor for Ingest Pipelines.

We have been using Elastic as a SIEM for nearly four years, and we have now spent the last ~15 months rolling it out as a Data Lake and Infrastructure Monitoring tool for the rest of our company. Over the years, we have come to rely quite heavily on Logstash for parsing and enrichment of all data being ingested (including Beats); nothing is written directly to Elasticsearch. The key enrichment aspects we rely on are Geo Location, DNS lookups, and appending to the related.* fields.

Now that Elastic Agent has been around for a while, I have been playing around with it on a few test devices as a centrally managed alternative to Beats agents; mainly Winlogbeat which is currently deployed to 5,000+ devices. Winlogbeat has always parsed each event at the agent (creating the ECS fields source.ip, user.name, etc.), so this allowed our Logstash enrichment on these ECS fields to work perfectly. Now that Elastic Agent uses Filebeat for Windows Event Logs, the parsing does not occur until hitting Ingest Pipelines at Elasticsearch. This unfortunately means the enrichment we have always done in Logstash is not possible.

I recently realized each Integration's Ingest Pipelines also run an *@custom pipeline after the pre-built pipeline runs. I thought this may be our way translate our Logstash pipelines to Ingest Pipelines and start using Elastic Agent with the same enrichment we have always done. I am unfortunately finding there is no DNS processor for Ingest Pipelines, and like I mentioned before, this is one thing we have come to rely on quite heavily.

I would like to request a DNS processor be created for Ingest Pipelines. The lack of a DNS processor for Ingest Pipelines is preventing us from using Elastic Agent.

Eric

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-data-management (Team:Data Management)

MakoWish commented 8 months ago

Got quite a few other customers that seem to want this, but no love from the Elastic side even after bringing it up with our CSM's. Was talking to @jamiehynds, and he pointed me to a similar issue he opened on the Integrations side: #2532

Erikg346 commented 5 months ago

+1 any update?