search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
41 stars 452 forks source link

[iptables]: not properly parsing VyOS logs #10881

Open buzzdeee opened 3 months ago

buzzdeee commented 3 months ago

Integration Name

Iptables [iptables]

Dataset Name

No response

Integration Version

1.16.1

Agent Version

8.15.0

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.0

OS Version and Architecture

Ubuntu 22.04.4 LTS

Software/API Version

No response

Error Message

No response

Event Original

[7598791.530254] [RULE_IN-2001-A] IN=eth0 OUT=eth1 MAC=00:1b:21:bc:c7:ba:52:54:00:4a:b3:d0:08:00 SRC=10.10.10.10 DST=10.11.11.11 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=5751 DF PROTO=UDP SPT=49704 DPT=53 LEN=64

What did you do?

I forward IPTables logs from a VyOS firewall to an agent where I use the Custom UDP input, with Syslog parsing enabled. Then in the custom ingestion pipeline for that dataset, I tell it to call the logs-iptables.log-1.16.1 pipeline.

What did you see?

some of these logs, at least some DROP logs, seem to be parsed fine with the UBIQUITY_LABEL, but others, at least what I saw now, ACCEPT or REJECT logs, have a "space" between the [UBIQUITY_LABEL] and the following IN=.... but the 4th grok pattern in the ingest pipeline Grok "Extracts values from "message" that match a grok pattern", looks alike: 

%{GREEDYDATA}\[%{UBIQUITI_LABEL}\]%{IPTABLES}%{SPACE}

due to the missing space, it didn't matched, and moved on with the more general pattern below. So I only ended up with event.action: drop events, and the others don't have that field set.

What did you expect to see?

Since VyOS more or less the same naming convention for its rules, the IPTables integration should parse them same way as Ubiquity IPTables logs. So that event.action will properly be set to accept, drop, reject.

Anything else?

updating above mentioned grok pattern to:

%{GREEDYDATA}\[%{UBIQUITI_LABEL}\]%{SPACE}?%{IPTABLES}%{SPACE}

did the trick for me. now in discover, I can see "drop" and "accept", for some reason, when there are reject rules I only see "r", haven't yet found where the event.action is expanded from the letters to the long names.

elasticmachine commented 3 months ago

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

buzzdeee commented 2 months ago

my proposed solution seem to have caused a redundant nested repeat operator error. Seems SPACE is implemented as \s and with that, the trailing ? caused errors, update to:

%{GREEDYDATA}\[%{UBIQUITI_LABEL}\]%{SPACE}%{IPTABLES}%{SPACE}