Agent | Local DNS Processing

[jamiehynds]

Many of our integrations including Palo Alto, Suricata (presumably all firewall/IDS integration) leverage the DNS processor within our ingest node pipelines. This is posing a problem for cloud customers who cannot enrich events (i.e. DNS reverse lookups) of on-prem hosts before shipping to the cloud.

Is moving the DNS processing from ingest node to the agent an option?

Providing optionality as to where processing takes place (regardless of processor) is a longer term goal, this particular is intended to focus solely on tackling the DNS issue.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

The Beats dns processor can be used in most integrations through the Advanced Settings by adding the YAML processors config. The problem is that in most cases the data hasn't been parsed so you don't have the IP(s) extracted to do the enrichment.

Is moving the DNS processing from ingest node to the agent an option?

One of the main reasons for doing this enrichment at the edge (e.g. in Agent/Beats) is to be able to communicate to internal DNS servers. For use cases that involve reverse lookups on internal addresses, this will only work if internal DNS server is accessible. In a lot of deployment scenarios Elasticsearch is running as a cloud service like Elastic Cloud.

In theory it could be implemented in Elasticsearch, but I wonder if that aligns with users architectures. I can't speak to whether Ingest Node would want to have a processor that needs to make network calls to enrich data (it doesn't do that today).

[matthiasledergerber]

I can just speak from our side, an DNS enrichment processor in the Ingest Pipeline of Elasticsearch would be a great contribution. We run the Elastic-Stack on-premise and there it is no problem to have DNS Lookups originating from the Elasticsearch Ingest Nodes. Other services running on these Servers need DNS too (APT Updates, Antivirus, etc,) so there is no real difference between having to do DNS Lookups from an Elasticsearch Ingest Pipeline or the Host system where Elasticsearch is installed itself.

I would even go as far as to say that DNS Enrichment handled by an Ingest Pipeline Processor is a Basic-Feature to be expected by such a great solution. :)

There are several use-cases where DNS Enrichment would be useful.

• Syslog Messages with only containing log.source.address as an IP-Address (Some Cisco IOS Devices do not send Hostname in the Syslog) -> Lookup IP to Hostname
• Enriching source.ip and destination.ip to further provide Information about the related systems. Often IP-Addresses do not be as useful for the user experience as Hostnames -> Lookup IP to Hostname
• Dropping Logstash because for us it's only needed to do DNS Enrichments -> Pushing Elasticsearch Ingest Pipelining further.

Features that would be great:

• Having the option of adding multiple DNS Resolvers, the first one answering should enrich the Log (Some Environments have separated DNS Servers for different Domains/Networks because of historical/security reasons)
• Option to add/remove the TLD from the DNS Answer
• Secure DNS Resolution

[willemdh]

It's crazy that this is not yet possible... Why would we want to move from Logstash to Elastic Agent if a simple reverse dns lookup appears to be impossible for datasets where the field which needs to be looked up is not yet available. For example in the Infoblox NIOS datastream, we would like to do a reverse lookup of the client.ip field. Using an Agent processor is not an option as the field client.ip is not yet available there...

[willemdh]

@jamiehynds @andrewkroh Could you please have a look at this issue. Created a support request (01383000) for this and the suggestion was to send the Infoblox data back to Logstash to parse the ip and then send back to es ingest.... We just come from Logstash and migrated to Elastic Agent... This seems rather inefficient to me.. Another suggestion in the support ticket was to use an enrich index for storing the dns lookup values............. Did Elastic support really just suggest I store all DNS info myself in an enrich index?? Please please allow us somehow to enrich our data with reverse dns lookups when using Elastic Agent.

[CannyGent]

Is there any update on this issue? It seems to me that Ingest Pipelines need this functionality as typically they are parsing out the IP address using grok etc. Can't believe have to go back to logstash to do this?

[willemdh]

Elastic Agent has multiple design issues atm.

• No DNS lookups possible....
• Very bad performance vs Filebeat (like 5 times as bad). The reason for the performance problems is: "Fleet-managed Elastic Agent does not support configuration of the internal memory queues by an end user."