Modify log value during decoding phase

[Nick-Honings]

Wazuh version	Component	Install type	Install method	Platform
X.Y.Z-rev	Wazuh component	Manager/Agent	Packages/Sources	OS version
4.1.5	Decoder	Manager	X	X

I have a feature request. I recently came across a problem when trying to correctly log and display DNS qnames. For example, i get the following qname from agents:

(3)www(6)google(3)com(0)

I would love to be able to somehow tell a decoder to replace the parentheses and the content within with a dot so it becomes:

www.google.com

I could try to implement this using the already available option with regex to extract the different components of the qname, but this might become tricky and seems a little time consuming. I think it might be handy to have a replace functionality in the decoder so it saves a little time.

Let me know what you think and if this request gives you enough information.

Best regards,

Nick Honings

[vikman90]

Hi @Nick-Honings,

Currently, Wazuh manager (Analysisd) merely interprets the input data and enriches it with some information about the environment (the agent and manager). However, we can achieve what you need using a replacement in Filebeat, as explained at #9888.

In this case, this is the replacement processor we can add to the pipeline (/usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json):

{ "gsub": { "field": "data.qname", "pattern": "^\\([0-9]+\\)", "replacement": "" } },
{ "gsub": { "field": "data.qname", "pattern": "\\(0\\)$", "replacement": "" } },
{ "gsub": { "field": "data.qname", "pattern": "\\([0-9]+\\)", "replacement": "." } }

The first statement removes the initial (...), the second one removes the last (0), and the last line replaces (...) with ..

After editing pipeline.json, restart Filebeat and reset the pipeline in Elasticsearch:

service filebeat restart
curl -XDELETE -k -u wazuh:X https://localhost:9200/_ingest/pipeline/filebeat-7.10.2-wazuh-alerts-pipeline

Nevertheless, pinging @wazuh/threat-intel to consider adding this feature in the upcoming decoder syntax redesign.

Hope it helps. Best regards.

[Nick-Honings]

Thank you for your reply. I hope the feature makes it into your product! In the mean time, I am going to try your suggestion.

[Nick-Honings]

I have tried your solution but it does not seem to work yet. The logs are still in the old format. Could you help me because I haven't been succesful in my attempts to fix it.

[vikman90]

Hi @Nick-Honings,

I've tested the pipeline change that I proposed and didn't see any problem. I did not remark it but that is an example replacement for strings in the alert's JSON field data.qname, assuming that the DNS qname is decoded into field qname. Just to rule out this detail, can you confirm that you renamed the field name?

Here is the Gsub processor reference at Elasticsearch documentation.

If this still does not work, please share a sample of an alert in JSON format here, so we will provide you with the correct pipeline change and will be able to test it.

[Nick-Honings]

In my decoder, the decoded value is named q_name, so it becomes data.q_name in the pipeline.

An example of a DNS log: 7/25/2021 9:49:07 PM 0B88 PACKET 0000014BD5331140 UDP Rcv 192.111.120.232 ca8a Q [0001 D NOERROR] A (3)www(6)google(3)com(0)

It is decoded with the following decoder:

`

\d+/\d+/\d+ \d+:\d+:\d+ \w+ \w+ PACKET

<regex>(\w+) (\w+) (\w+) (\d+.\d+.\d+.\d+)\s+(\w+) (\.) (\S) [(\w+) (\.) (\w+\s+\w+)] (\w+)\s+(\S+)</regex>
<order>packet_id,protocol,snd_rcv,remote_ip,xid,query,opcode,hex_flags,char_code,resp_code,q_type,q_name</order>

`

When I test it with the Wazuh Dev console (PUT /logtest) I can confirm it has not replaced the values specified in the gsub processor.

[vikman90]

@Nick-Honings apologies for the delay.

I'm afraid that the log testing won't transform the data. As I mentioned, the Wazuh Manager does not implement log value modification. Analysisd (and its testing subcomponent Logtest) runs log decoding and rule matching. The resulting alert will contain the same values as were given in the input. That's why we will consider this issue as a feature request.

On the other hand, though we cannot currently modify values, we can set up Elasticsearch to do it when ingesting alerts. So even when Wazuh will produce an alert having a field data.q_name with the undesired format, Elasticsearch will transform it, and Kibana should show the alerts as you want.

Please let us know if this does not happen.

Hope that helps. Best regards.

[Nick-Honings]

@vikman90 I managed to get it working correctly. The values are now correctly displayed in Kibana. Thank you for that! However, Wazuh now misses all other alerts. Does this have something to do with the gsub processor? If so, where should I place it to make it work?

I attached my pipeline.json below.

` I am sorry if I missed something in your explanation.

[Nick-Honings]

I managed to fix it by looking at the ElasticSearch documentation. Thank you very much for your help!

[vikman90]

@Nick-Honings I'm glad you got it!

We will keep you posted when we start to develop this enhancement. Thank you for your proposal.

Best regards.