Non-TCP/UDP traffic - Githubissues

Thank you for creating the Graylog Grok pattern and posting it. I just started working with Graylog and grok, and your extractor is very helpful. I have three comments:

1) The extractor only works for me if I check the rfc5424 option in OPNsense System->Logging->Remote->feed settings. It took me a while to figure this out and I suggest adding it to readme. 2) The end of the pattern does not seem to capture all of the TCP fields. See here for the field definitions: https://github.com/opnsense/ports/blob/master/opnsense/filterlog/files/description.txt I changed the pattern ending to: (,%{DATA:tcp_flags},%{INT:tcp_seq},%{DATA:tcp_ack},%{INT:tcp_windows},%{DATA:tcp_urg},%{DATA:tcp_options})?%{GREEDYDATA:other} 3) The extractor does not extract fields from messages that are not TCP/UDP. For example ICMP traffic. I created a second extractor to handle non TCP/UDP messages: <%{INT:syslog_priority}>%{INT} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} filterlog %{INT:pid} - \[meta sequenceId="%{INT:sequence_id}"\] %{INT:rule_number},%{DATA:sub_rule_number},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},match,%{WORD:action},%{WORD:direction},%{INT:ip_version},%{DATA:tos},%{DATA:ecn},%{INT:ttl},%{INT:id},%{INT:offset},%{DATA:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IPV4:src_ip},%{IPV4:dst_ip},datalength=%{INT:data_length}%{GREEDYDATA:other} 4) I am using the %{GREEDYDATA:other} field at the end to potentially catch some changes to the source data structure. I am not sure if it is needed or will work.

After making the changes above, for the last hour I have 100% of the log messages being parsed. I am using OPN 24.7.4 and Graylog 6.0.6. I am not using IPv6 or CARP so I don't know how my changes will work with those messages.

I'm still learning, so if I've missed something please let me know. I am happy to collaborate on any updates. Thanks again for putting this helpful code out there.

secdoc / OPNsense-24.7-Graylog-Grok-Patterns

Non-TCP/UDP traffic #1