search

pfelk / pfelk

pfSense/OPNsense + Elastic Stack
https://pfelk.github.io/pfelk/
Other
1.07k stars 191 forks source link

field [observer][ip] being received as "gateway" instead of the actual IP address #284

Closed colinator19 closed 2 years ago

colinator19 commented 3 years ago

Describe the bug Still working on setting up logging via syslog-ng (sending logs encrypted) and Suricata is working perfectly however the firewall logs are not. I have set syslog-ng to read the log files (dhcpd.log and filter.log and then send them to logstash. Logstash is receiving them correctly however is unable to process the entry.

Host IP is set to "gateway" rather then IP resulting in Logstash not able to process the log entry for DHCPD and Filterlog. I have no idea where the "gateway" is coming from, it is not the hostname of PfSense and it is nowhere set up either. It is not present in the logstash config files so not sure why it is set to "gateway" rather then the IP.

It is worth mentioning that Suricata has the same problem but it does not error out on it and appears to be working fine. I did however manually put in this line within the app filter for suricata to replace the hostname from "gateway" to the actual hostname found in the log entry. This will allow me to filter within the dashboard when using multiple firewalls. replace => { "[observer][hostname]" => "%{[host][name]}" }

Error message in Logstash: [WARN ] 2021-03-07 13:23:53.003 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"pfelk-firewall-2021.03", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5ab2907a>], :response=>{"index"=>{"_index"=>"pfelk-firewall-2021.03", "_type"=>"_doc", "_id"=>"BBvbDHgBplMWZIE6xo8o", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [observer.ip] of type [ip] in document with id 'BBvbDHgBplMWZIE6xo8o'. Preview of field's value: 'gateway'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'gateway' is not an IP string literal."}}}}}

This is the source config in syslog-ng: { wildcard-file( base-dir("/var/log/") filename-pattern("filter.log") recursive(yes) follow-freq(1) program-override("filterlog") flags(no-parse) ); };

Grok filter: filter { if [observer][type] == "firewall" or [observer][type] == "haproxy" or [observer][type] == "suricata" { grok { match => {"message" => "%{POSINT:[log][syslog][priority]}?(%{INT:[log][syslog][version]}\s*)?(%{SYSLOGTIMESTAMP:[event][created]}|%{TIMESTAMP_ISO8601:[event][created]})\s(%{SYSLOGHOST:[host][name]}\s+)?%{PROG:[process][name]}\s*?(\[)?%{POSINT:[process][pid]}(\]:)?\s*(\-\s*\-)?\s*%{GREEDYDATA:filter_message}|%{POSINT:[log][syslog][priority]}?(%{INT:[log][syslog][version]}\s*)?(%{SYSLOGTIMESTAMP:[event][created]}|%{TIMESTAMP_ISO8601:[event][created]})\s(%{SYSLOGHOST:[host][name]}\s+)?%{PROG:[process][name]}\:\s%{GREEDYDATA:filter_message}"} } date { match => [ "[event][created]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] target => "[event][created]" } } }

App filter: if [process][name] =~ /^filterlog$/ { mutate { add_tag => [ "firewall" ] add_field => { "[ecs][version]" => "1.7.0" } add_field => { "[event][dataset]" => "pfelk.firewall" } } grok { patterns_dir => [ "/etc/pfelk/patterns" ] match => [ "filter_message", "%{PF_LOG_ENTRY}" ] } }

Firewall System (please complete the following information):

Elasticsearch, Logstash, Kibana (please complete the following information):

a3ilson commented 3 years ago

Glad to see you using syslog-ng...and wish pfSense would make it the default. There is no value defined within pfelk that equals gateway. I'd be happy to assist but will likely need to see the original logs to better assist.

colinator19 commented 3 years ago

Thanks for trying to assist and I could not agree more syslog-ng should be default on pfSense. I have a feeling that a reverse DNS might be playing a role here somewhere which replaces the IP for the hostname although my firewall is not known as 'gateway' in my own DNS. This value is at least nowhere to be found in the source logs so either syslog-ng does something or logstash does....

Since it appears to be present in all logs that I am trying to process, I think the cause will be the same so just putting the dhcpd logs from pfSense and the logstash logs below. Please let me know if you need anything specific and I'll grab it too.

(note that I did replace IP's and MAC details with fake data) dhcpd log from pfSense: Mar 8 19:12:00 vFW01 newsyslog[75529]: logfile turned over due to size>500K Mar 8 19:12:21 vFW01 dhcpd[30057]: reuse_lease: lease age 1574 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:12:21 vFW01 dhcpd[30057]: DHCPDISCOVER from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:21 vFW01 dhcpd[30057]: DHCPOFFER on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:21 vFW01 dhcpd[30057]: reuse_lease: lease age 1574 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:12:21 vFW01 dhcpd[30057]: DHCPREQUEST for 172.16.1.10 (172.16.1.1) from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:21 vFW01 dhcpd[30057]: DHCPACK on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:41 vFW01 dhcpd[30057]: reuse_lease: lease age 1594 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:12:41 vFW01 dhcpd[30057]: DHCPDISCOVER from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:41 vFW01 dhcpd[30057]: DHCPOFFER on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:41 vFW01 dhcpd[30057]: reuse_lease: lease age 1594 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:12:41 vFW01 dhcpd[30057]: DHCPREQUEST for 172.16.1.10 (172.16.1.1) from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:12:41 vFW01 dhcpd[30057]: DHCPACK on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:07 vFW01 dhcpd[30057]: reuse_lease: lease age 1620 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:13:07 vFW01 dhcpd[30057]: DHCPDISCOVER from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:07 vFW01 dhcpd[30057]: DHCPOFFER on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:07 vFW01 dhcpd[30057]: reuse_lease: lease age 1620 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:13:07 vFW01 dhcpd[30057]: DHCPREQUEST for 172.16.1.10 (172.16.1.1) from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:07 vFW01 dhcpd[30057]: DHCPACK on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:27 vFW01 dhcpd[30057]: reuse_lease: lease age 1640 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:13:27 vFW01 dhcpd[30057]: DHCPDISCOVER from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:27 vFW01 dhcpd[30057]: DHCPOFFER on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:27 vFW01 dhcpd[30057]: reuse_lease: lease age 1640 (secs) under 25% threshold, reply with unaltered, existing lease for 172.16.1.10 Mar 8 19:13:27 vFW01 dhcpd[30057]: DHCPREQUEST for 172.16.1.10 (172.16.1.1) from aa:aa:aa:aa:aa:aa via mlxen1.20 Mar 8 19:13:27 vFW01 dhcpd[30057]: DHCPACK on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20

Syslog-NG logs: Mar 8 16:50:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=740478', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615218600', processed='destination(_DEFAULT)=201', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=201', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=740478', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=740478', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=35719', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615218590', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=477', processed='src.file(filterlog#0,/var/log/filter.log)=86150', stamp='src.file(filterlog#0,/var/log/filter.log)=1615218589', processed='destination(Suricata)=740478', processed='center(queued)=862749', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=201', stamp='src.internal(_DEFAULT#0)=1615218000', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=122070', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=122070', processed='source(Suricata)=740478', processed='destination(syslog-ng)=122070', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=862548', processed='source(filterlog)=86150', processed='source(dhcpd)=35719', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 17:00:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=744549', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615219199', processed='destination(_DEFAULT)=202', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=202', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=744549', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=744549', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=35891', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615219196', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=480', processed='src.file(filterlog#0,/var/log/filter.log)=86329', stamp='src.file(filterlog#0,/var/log/filter.log)=1615219199', processed='destination(Suricata)=744549', processed='center(queued)=867173', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=202', stamp='src.internal(_DEFAULT#0)=1615218600', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=122422', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=122422', processed='source(Suricata)=744549', processed='destination(syslog-ng)=122422', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=866971', processed='source(filterlog)=86329', processed='source(dhcpd)=35891', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 17:10:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=748104', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615219799', processed='destination(_DEFAULT)=203', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=203', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=748104', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=748104', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=36071', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615219795', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=483', processed='src.file(filterlog#0,/var/log/filter.log)=86508', stamp='src.file(filterlog#0,/var/log/filter.log)=1615219788', processed='destination(Suricata)=748104', processed='center(queued)=871089', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=203', stamp='src.internal(_DEFAULT#0)=1615219200', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=122782', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=122782', processed='source(Suricata)=748104', processed='destination(syslog-ng)=122782', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=870886', processed='source(filterlog)=86508', processed='source(dhcpd)=36071', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 17:20:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=752611', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615220400', processed='destination(_DEFAULT)=204', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=204', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=752611', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=752611', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=36239', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615220395', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=486', processed='src.file(filterlog#0,/var/log/filter.log)=86692', stamp='src.file(filterlog#0,/var/log/filter.log)=1615220398', processed='destination(Suricata)=752611', processed='center(queued)=875950', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=204', stamp='src.internal(_DEFAULT#0)=1615219800', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=123135', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=123135', processed='source(Suricata)=752611', processed='destination(syslog-ng)=123135', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=875746', processed='source(filterlog)=86692', processed='source(dhcpd)=36239', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 17:30:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=756274', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615220999', processed='destination(_DEFAULT)=205', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=205', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=756274', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=756274', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=36399', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615220996', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=489', processed='src.file(filterlog#0,/var/log/filter.log)=86859', stamp='src.file(filterlog#0,/var/log/filter.log)=1615220989', processed='destination(Suricata)=756274', processed='center(queued)=879942', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=205', stamp='src.internal(_DEFAULT#0)=1615220400', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=123463', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=123463', processed='source(Suricata)=756274', processed='destination(syslog-ng)=123463', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=879737', processed='source(filterlog)=86859', processed='source(dhcpd)=36399', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 17:40:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=759557', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615221600', processed='destination(_DEFAULT)=206', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=206', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=759557', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=759557', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=36573', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615221576', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=492', processed='src.file(filterlog#0,/var/log/filter.log)=87002', stamp='src.file(filterlog#0,/var/log/filter.log)=1615221584', processed='destination(Suricata)=759557', processed='center(queued)=883544', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=206', stamp='src.internal(_DEFAULT#0)=1615221000', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=123781', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=123781', processed='source(Suricata)=759557', processed='destination(syslog-ng)=123781', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=883338', processed='source(filterlog)=87002', processed='source(dhcpd)=36573', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 17:48:01 vFW01 syslog-ng[49845]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/var/log/filter.log)', stored_inode='481715', cur_file_inode='481606', stored_size='77', cur_file_size='77', raw_stream_pos='514767' Mar 8 17:50:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=762863', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615222200', processed='destination(_DEFAULT)=208', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=208', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=762863', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=762863', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=36743', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615222200', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=495', processed='src.file(filterlog#0,/var/log/filter.log)=87218', stamp='src.file(filterlog#0,/var/log/filter.log)=1615222197', processed='destination(Suricata)=762863', processed='center(queued)=887240', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=208', stamp='src.internal(_DEFAULT#0)=1615222081', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=124169', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=124169', processed='source(Suricata)=762863', processed='destination(syslog-ng)=124169', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=887032', processed='source(filterlog)=87218', processed='source(dhcpd)=36743', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:00:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=767017', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615222800', processed='destination(_DEFAULT)=209', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=209', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=767017', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=767017', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=36915', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615222800', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=498', processed='src.file(filterlog#0,/var/log/filter.log)=87652', stamp='src.file(filterlog#0,/var/log/filter.log)=1615222799', processed='destination(Suricata)=767017', processed='center(queued)=892002', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=209', stamp='src.internal(_DEFAULT#0)=1615222200', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=124776', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=124776', processed='source(Suricata)=767017', processed='destination(syslog-ng)=124776', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=891793', processed='source(filterlog)=87652', processed='source(dhcpd)=36915', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:10:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=770874', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615223400', processed='destination(_DEFAULT)=210', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=210', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=770874', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=770874', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=37097', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615223399', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=501', processed='src.file(filterlog#0,/var/log/filter.log)=88036', stamp='src.file(filterlog#0,/var/log/filter.log)=1615223399', processed='destination(Suricata)=770874', processed='center(queued)=896427', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=210', stamp='src.internal(_DEFAULT#0)=1615222800', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=125343', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=125343', processed='source(Suricata)=770874', processed='destination(syslog-ng)=125343', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=896217', processed='source(filterlog)=88036', processed='source(dhcpd)=37097', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:20:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=775069', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615223999', processed='destination(_DEFAULT)=211', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=211', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=775069', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=775069', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=37263', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615223999', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=504', processed='src.file(filterlog#0,/var/log/filter.log)=88394', stamp='src.file(filterlog#0,/var/log/filter.log)=1615223999', processed='destination(Suricata)=775069', processed='center(queued)=901148', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=211', stamp='src.internal(_DEFAULT#0)=1615223400', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=125868', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=125868', processed='source(Suricata)=775069', processed='destination(syslog-ng)=125868', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=900937', processed='source(filterlog)=88394', processed='source(dhcpd)=37263', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:30:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=778841', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615224599', processed='destination(_DEFAULT)=212', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=212', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=778841', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=778841', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=37429', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615224598', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=507', processed='src.file(filterlog#0,/var/log/filter.log)=88745', stamp='src.file(filterlog#0,/var/log/filter.log)=1615224599', processed='destination(Suricata)=778841', processed='center(queued)=905439', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=212', stamp='src.internal(_DEFAULT#0)=1615224000', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=126386', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=126386', processed='source(Suricata)=778841', processed='destination(syslog-ng)=126386', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=905227', processed='source(filterlog)=88745', processed='source(dhcpd)=37429', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:40:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=783104', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615225200', processed='destination(_DEFAULT)=213', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=213', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=783104', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=783104', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=37601', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615225199', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=510', processed='src.file(filterlog#0,/var/log/filter.log)=89491', stamp='src.file(filterlog#0,/var/log/filter.log)=1615225199', processed='destination(Suricata)=783104', processed='center(queued)=910622', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=213', stamp='src.internal(_DEFAULT#0)=1615224600', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=127305', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=127305', processed='source(Suricata)=783104', processed='destination(syslog-ng)=127305', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=910409', processed='source(filterlog)=89491', processed='source(dhcpd)=37601', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:50:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=787089', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615225800', processed='destination(_DEFAULT)=214', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=214', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=787089', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=787089', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=37766', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615225775', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=513', processed='src.file(filterlog#0,/var/log/filter.log)=89927', stamp='src.file(filterlog#0,/var/log/filter.log)=1615225800', processed='destination(Suricata)=787089', processed='center(queued)=915210', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=214', stamp='src.internal(_DEFAULT#0)=1615225200', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=127907', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=127907', processed='source(Suricata)=787089', processed='destination(syslog-ng)=127907', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=914996', processed='source(filterlog)=89927', processed='source(dhcpd)=37766', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 18:53:01 vFW01 syslog-ng[49845]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/var/log/filter.log)', stored_inode='481606', cur_file_inode='481764', stored_size='77', cur_file_size='77', raw_stream_pos='516174' Mar 8 19:00:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=791116', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615226400', processed='destination(_DEFAULT)=216', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=216', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=791116', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=791116', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=37935', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615226377', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=516', processed='src.file(filterlog#0,/var/log/filter.log)=90541', stamp='src.file(filterlog#0,/var/log/filter.log)=1615226400', processed='destination(Suricata)=791116', processed='center(queued)=920024', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=216', stamp='src.internal(_DEFAULT#0)=1615225981', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=128692', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=128692', processed='source(Suricata)=791116', processed='destination(syslog-ng)=128692', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=919808', processed='source(filterlog)=90541', processed='source(dhcpd)=37935', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 19:10:00 vFW01 syslog-ng[49845]: Log statistics; processed='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=795812', stamp='src.file(Suricata#0,/var/log/suricata/suricata_mlxen17060/eve.json)=1615227000', processed='destination(_DEFAULT)=217', dropped='global(internal_source)=0', queued='global(internal_source)=0', processed='global(sdata_updates)=0', processed='source(_DEFAULT)=217', dropped='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=795812', queued='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', written='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=795812', truncated_bytes='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='global(msg_clones)=0', truncated_count='dst.tcp(Suricata#0,tcp,192.168.1.5:5040)=0', processed='src.file(dhcpd#0,/var/log/dhcpd.log)=38118', stamp='src.file(dhcpd#0,/var/log/dhcpd.log)=1615226977', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=519', processed='src.file(filterlog#0,/var/log/filter.log)=90811', stamp='src.file(filterlog#0,/var/log/filter.log)=1615226993', processed='destination(Suricata)=795812', processed='center(queued)=925175', queued='global(scratch_buffers_bytes)=0', processed='src.internal(_DEFAULT#0)=217', stamp='src.internal(_DEFAULT#0)=1615226400', dropped='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=129146', queued='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', written='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=129146', processed='source(Suricata)=795812', processed='destination(syslog-ng)=129146', truncated_bytes='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='center(received)=924958', processed='source(filterlog)=90811', processed='source(dhcpd)=38118', truncated_count='dst.tcp(syslog-ng#0,tcp,192.168.1.5:5141)=0', processed='global(internal_queue_length)=0' Mar 8 19:12:01 vFW01 syslog-ng[49845]: The current log file has a mismatching size/inode information, restarting from the beginning; state='affile_sd_curpos(/var/log/dhcpd.log)', stored_inode='481760', cur_file_inode='481606', stored_size='761', cur_file_size='77', raw_stream_pos='511242'

(note: for Suricata I have adjusted the grok filter to use the hostname found in the logs which replaces 'gateway', this is why the hostname in logstash seems correct for the Suricata logs. without this adjustment the hostname is shown as gateway in logstash as well) Logstash log: { "@version" => "1", "@timestamp" => 2021-03-08T18:20:43.336Z, "observer" => { "serial_number" => "001", "product" => "Supermicro", "hostname" => "vFW01", "type" => "suricata", "name" => "IDS" }, "tags" => [ [0] "suricata", [1] "IP_Private_Source", [2] "IP_Private_Destination" ], "source" => { "ip" => "192.168.1.22", "port" => "55724" }, "destination" => { "ip" => "172.16.1.201", "port" => "80" }, "process" => { "name" => "suricata" }, "suricata" => { "eve" => { "proto" => "TCP", "flow_id" => 1008172353077766, "in_iface" => "mlxen1", "vlan" => [ [0] 20 ], "http" => { "accept" => "application/json, text/plain, */*", "length" => 877, "url" => "/json/si", "connection" => "close", "content_length" => "877", "http_content_type" => "application/json", "hostname" => "172.16.1.201", "status" => 200, "content_type" => "application/json", "accept_encoding" => "gzip, deflate", "http_user_agent" => "PythonWLED/0.4.4", "protocol" => "HTTP/1.1", "http_method" => "GET" }, "dest_port" => 80, "src_port" => 55724, "timestamp" => "2021-03-08T19:20:43.122475+0100", "dest_ip" => "172.16.1.201", "event_type" => "http", "tx_id" => 0, "src_ip" => "192.168.1.22" } }, "port" => 36992, "event" => { "dataset" => "pfelk.suricata", "original" => "<13>Mar 8 19:20:43 vFW01 suricata: {\"timestamp\":\"2021-03-08T19:20:43.122475+0100\",\"flow_id\":1008172353077766,\"in_iface\":\"mlxen1\",\"event_type\":\"http\",\"vlan\":[20],\"src_ip\":\"192.168.1.22\",\"src_port\":55724,\"dest_ip\":\"172.16.1.201\",\"dest_port\":80,\"proto\":\"TCP\",\"tx_id\":0,\"http\":{\"hostname\":\"172.16.1.201\",\"url\":\"\\/json\\/si\",\"http_user_agent\":\"PythonWLED\\/0.4.4\",\"http_content_type\":\"application\\/json\",\"accept\":\"application\\/json, text\\/plain, *\\/*\",\"accept_encoding\":\"gzip, deflate\",\"connection\":\"close\",\"content_length\":\"877\",\"content_type\":\"application\\/json\",\"http_method\":\"GET\",\"protocol\":\"HTTP\\/1.1\",\"status\":200,\"length\":877}}", "created" => 2021-03-08T19:20:43.000Z } } { "@version" => "1", "@timestamp" => 2021-03-08T18:20:43.336Z, "observer" => { "serial_number" => "001", "product" => "Supermicro", "hostname" => "vFW01", "type" => "suricata", "name" => "IDS" }, "tags" => [ [0] "suricata", [1] "IP_Private_Source", [2] "IP_Private_Destination" ], "source" => { "ip" => "172.16.1.201", "port" => "80" }, "destination" => { "ip" => "192.168.1.22", "port" => "55724" }, "process" => { "name" => "suricata" }, "suricata" => { "eve" => { "proto" => "TCP", "flow_id" => 1008172353077766, "app_proto" => "http", "in_iface" => "mlxen1", "vlan" => [ [0] 20 ], "http" => { "http_content_type" => "application/json", "hostname" => "172.16.1.201", "status" => 200, "url" => "/json/si", "length" => 877, "http_user_agent" => "PythonWLED/0.4.4", "protocol" => "HTTP/1.1", "http_method" => "GET" }, "dest_port" => 55724, "src_port" => 80, "timestamp" => "2021-03-08T19:20:43.122475+0100", "dest_ip" => "192.168.1.22", "fileinfo" => { "size" => 877, "gaps" => false, "sid" => [], "stored" => false, "state" => "CLOSED", "filename" => "/json/si", "tx_id" => 0 }, "event_type" => "fileinfo", "src_ip" => "172.16.1.201" } }, "port" => 36992, "event" => { "dataset" => "pfelk.suricata", "original" => "<13>Mar 8 19:20:43 vFW01 suricata: {\"timestamp\":\"2021-03-08T19:20:43.122475+0100\",\"flow_id\":1008172353077766,\"in_iface\":\"mlxen1\",\"event_type\":\"fileinfo\",\"vlan\":[20],\"src_ip\":\"172.16.1.201\",\"src_port\":80,\"dest_ip\":\"192.168.1.22\",\"dest_port\":55724,\"proto\":\"TCP\",\"http\":{\"hostname\":\"172.16.1.201\",\"url\":\"\\/json\\/si\",\"http_user_agent\":\"PythonWLED\\/0.4.4\",\"http_content_type\":\"application\\/json\",\"http_method\":\"GET\",\"protocol\":\"HTTP\\/1.1\",\"status\":200,\"length\":877},\"app_proto\":\"http\",\"fileinfo\":{\"filename\":\"\\/json\\/si\",\"sid\":[],\"gaps\":false,\"state\":\"CLOSED\",\"stored\":false,\"size\":877,\"tx_id\":0}}", "created" => 2021-03-08T19:20:43.000Z } } [WARN ] 2021-03-08 18:20:43.501 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"pfelk-firewall-2021.03", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x18c4fbb3>], :response=>{"index"=>{"_index"=>"pfelk-firewall-2021.03", "_type"=>"_doc", "_id"=>"R0cRE3gBplMWZIE65udt", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [observer.ip] of type [ip] in document with id 'R0cRE3gBplMWZIE65udt'. Preview of field's value: 'gateway'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'gateway' is not an IP string literal."}}}}} [WARN ] 2021-03-08 18:20:43.502 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"pfelk-firewall-2021.03", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x6ca8592a>], :response=>{"index"=>{"_index"=>"pfelk-firewall-2021.03", "_type"=>"_doc", "_id"=>"SEcRE3gBplMWZIE65udt", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [observer.ip] of type [ip] in document with id 'SEcRE3gBplMWZIE65udt'. Preview of field's value: 'gateway'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'gateway' is not an IP string literal."}}}}}

a3ilson commented 3 years ago

Thanks!

I only did a quick cursory review and only noted Suricata and DHCP logs above.

DHCP logs are in a consistent format but not the format in which the filters will successfully parse. This is likely due to utilizing syslog-ng on pfSense. This works fine on OPNsense where syslog-ng is the default but it has not been fully developed to support pfSense.

Example: <190>Mar 8 13:33:50 firewall.domain.com dhcpd[44757]: DHCPREQUEST for 192.168.200.41 from 00:4c:e3:01:aa:ff via igb1_vlan500

Where as your logs are in this format: Mar 8 19:13:07 vFW01 dhcpd[30057]: DHCPACK on 172.16.1.10 to aa:aa:aa:aa:aa:aa via mlxen1.20

Suricata. I haven't updated the wiki yet but issue #276 was the troubleshooting for getting suricata to work on pfSense. This stemmed from an earlier issue where pfSense truncated the messages via issue #111

To get it working, I'll need other logs if you are planning on logging more than DHCP and believe the method for Suricata (pending the wiki update) will suffice. I'll need to build out revised filters for utilization and require your assistance to test. Let me know if that works.

colinator19 commented 3 years ago

I did see #276 and did update the grok filter specified there which did allow me to process the Suricata logs fine with the exception on the hostname being 'gateway'. I did resolve this by adding this replace => { "[observer][hostname]" => "%{[host][name]}" } so it will use the actual hostname noted in the log of my firewall when putting the data into Elastic. This only works for Suricata as it is expecting either the hostname OR ip but it looks like it needs to be IP only for the firewall logs as it is not able to process a hostname value.

I would be happy to help testing so please let me know if I need to try anything. In the meantime I am trying to look at a possible reverse DNS that might be causing the issue.

I do not use all the logs and/or services but I have added the other pfSense logs I am looking to use below as well; filter.log: Mar 8 20:01:12 vFW01 filterlog[25715]: 50,,,1000000118,mlxen0.300,match,block,in,4,0x0,,53,26671,0,DF,6,tcp,1452,1.8.5.30,192.168.1.8,51151,28967,1400,A,287368985:287370385,3832690971,501,,nop;nop;TS Mar 8 20:01:16 vFW01 filterlog[25715]: 50,,,1000000118,mlxen0.300,match,block,in,4,0x0,,56,13349,0,DF,6,tcp,60,1.6.5.164,192.168.1.8,47374,28967,0,S,1197942397,,64240,,mss;sackOK;TS;nop;wscale Mar 8 20:01:17 vFW01 filterlog[25715]: 4,,,1000000103,mlxen0.300,match,block,in,4,0x0,,247,23241,0,none,6,tcp,40,15.68.2.207,31.2.24.1,49918,43385,0,S,2557219796,,1024,, Mar 8 20:01:17 vFW01 filterlog[25715]: 51,,,1000000119,mlxen1,match,block,in,4,0x0,,64,2032,0,DF,6,tcp,60,192.168.1.130,1.8.5.165,45700,443,0,S,2904944000,,64240,,mss;sackOK;TS;nop;wscale Mar 8 20:01:18 vFW01 filterlog[25715]: 50,,,1000000118,mlxen0.300,match,block,in,4,0x0,,56,59198,0,DF,6,tcp,60,172.99.65.247,192.168.1.8,50380,28967,0,S,4202586047,,65535,,mss;sackOK;TS;nop;wscale Mar 8 20:01:19 vFW01 filterlog[25715]: 50,,,1000000118,mlxen0.300,match,block,in,4,0x0,,56,22959,0,DF,6,tcp,60,1.6.5.164,192.168.1.8,48868,28967,0,S,2375548345,,64240,,mss;sackOK;TS;nop;wscale Mar 8 20:01:25 vFW01 filterlog[25715]: 4,,,1000000103,mlxen0.300,match,block,in,4,0x0,,252,21595,0,none,6,tcp,40,92.6.1.97,31.2.24.1,59660,7260,0,S,2231937123,,1024,, Mar 8 20:01:25 vFW01 filterlog[25715]: 51,,,1000000119,mlxen1,match,block,in,4,0x0,,64,61650,0,DF,6,tcp,60,192.168.1.130,1.8.5.165,45510,443,0,S,278096645,,64240,,mss;sackOK;TS;nop;wscale Mar 8 20:01:26 vFW01 filterlog[25715]: 51,,,1000000119,mlxen1.40,match,block,in,4,0x0,,64,8146,0,DF,6,tcp,52,192.168.1.8,1.8.5.30,28967,51151,0,A,,287368985,83,,nop;nop;TS Mar 8 20:01:26 vFW01 filterlog[25715]: 50,,,1000000118,mlxen0.300,match,block,in,4,0x4,,53,1857,0,DF,6,tcp,1091,9.2.18.195,192.168.1.8,16784,28967,1051,PA,3609515210:3609516261,3323192629,67,, Mar 8 20:01:27 vFW01 filterlog[25715]: 4,,,1000000103,mlxen0.300,match,block,in,4,0x0,,55,35105,0,DF,6,tcp,60,8.21.10.52,31.2.24.1,59466,14000,0,S,2480979146,,29200,,mss;sackOK;TS;nop;wscale

ntpd.log: Mar 8 19:38:02 vFW01 ntpd[36565]: ---------------------------------------------------- Mar 8 19:38:02 vFW01 ntpd[36565]: ntp-4 is maintained by Network Time Foundation, Mar 8 19:38:02 vFW01 ntpd[36565]: Inc. (NTF), a non-profit 501(c)(3) public-benefit Mar 8 19:38:02 vFW01 ntpd[36565]: corporation. Support and training for ntp-4 are Mar 8 19:38:02 vFW01 ntpd[36565]: available at https://www.nwtime.org/support Mar 8 19:38:02 vFW01 ntpd[36565]: ---------------------------------------------------- Mar 8 19:38:02 vFW01 ntpd[36812]: proto: precision = 7.830 usec (-17) Mar 8 19:38:02 vFW01 ntpd[36812]: basedate set to 2021-01-24 Mar 8 19:38:02 vFW01 ntpd[36812]: gps base set to 2021-01-24 (week 2142) Mar 8 19:38:02 vFW01 ntpd[36812]: Listen and drop on 0 v6wildcard [::]:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 2 lo0 [::1]:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 3 lo0 [fe80::1%2]:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 4 lo0 127.0.0.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 7 mlxen1 192.168.1.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 9 mlxen1.20 172.16.20.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 11 mlxen1.40 172.16.40.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 13 mlxen1.50 172.16.50.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 15 mlxen1.60 172.16.60.1.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 17 mlxen0.300 3.2.183.101:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listen normally on 18 wg0 192.168.90.1:123 Mar 8 19:38:02 vFW01 ntpd[36812]: Listening on routing socket on fd #40 for interface updates Mar 8 19:38:02 vFW01 ntpd[36812]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized Mar 8 19:38:02 vFW01 ntpd[36812]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized Mar 8 19:38:13 vFW01 ntpd[36812]: Soliciting pool server 202.12.97.45 Mar 8 19:38:14 vFW01 ntpd[36812]: Soliciting pool server 201.217.3.86 Mar 8 19:38:15 vFW01 ntpd[36812]: Soliciting pool server 209.115.181.108 Mar 8 19:38:16 vFW01 ntpd[36812]: Soliciting pool server 45.33.103.94

colinator19 commented 3 years ago

It seems that because of the IP error, logstash is still forwarding the logs to Elastic but into another index "logstash-2021.03.08", I think that might be default behavior since it cannot put the data into "pfelk-firewall-2021.03". Below I have added a record found in that index where the observer.ip field is still 'gateway'....

I am looking at syslog-ng options to see if this might help, so far tried the below options with no luck yet; { chain_hostnames(no); check_hostname (yes); keep_hostname (no); use_fqdn (no); use_dns (no); };

Log record in Logstash: observer.type:firewall @version:1 @timestamp:Mar 8, 2021 @ 20:39:51.517 observer.serial_number:001 observer.product:Supermicro observer.ip:gateway observer.name:pfSense pf.transport.data_length:0 pf.tcp.window:64240 pf.tcp.flags:S pf.tcp.options:mss, sackOK, TS, nop, wscale pf.tcp.sequence_number:1239800435 pf.packet.length:60 pf.ipv4.ttl:56 pf.ipv4.offset:0 pf.ipv4.tos:0x0 pf.ipv4.packet.id:6236 pf.ipv4.flags:DF ecs.version:1.7.0 tags:firewall, _geoip_lookup_failure, GeoIP_Source, IP_Private_Destination rule.ruleset:50 rule.uuid:1000000118 rule.alias:50 rule.description:mlxen0.300: 50 source.geo.timezone:Europe/Helsinki source.geo.latitude:60.171 source.geo.country_code3:FI source.geo.country_name:Finland source.geo.location.lon:24.938 source.geo.location.lat:60.171 source.geo.longitude:24.938 source.geo.country_iso_code:FI source.geo.ip:9.21.28.14 source.geo.continent_code:EU source.port:53294 source.ip:9.21.28.14 destination.ip:192.168.1.8 destination.port:28967 network.transport:tcp network.direction:in network.type:4 network.name:mlxen0.300 network.iana_number:6 process.name:filterlog port:64,942 interface.alias:mlxen0.300 interface.name:mlxen0.300 event.dataset:pfelk.firewall event.reason:match event.original:<13>Mar 8 20:39:51 vFW01 filterlog: Mar 8 20:39:51 vFW01 filterlog[25715]: 50,,,1000000118,mlxen0.300,match,block,in,4,0x0,,56,6236,0,DF,6,tcp,60,9.21.28.14,192.168.1.8,53294,28967,0,S,1239800435,,64240,,mss;sackOK;TS;nop;wscale event.action:block event.created:Mar 8, 2021 @ 21:39:51.000 _id:XUpaE3gBplMWZIE6WRfF _type:_doc _index:logstash-2021.03.08 _score: -

revere521 commented 3 years ago

you should be able to hard code the ip of pfsense with the host override in syslog NG

with just something like: host-override("192.168.1.1")

you would put that in your source statement for each one

revere521 commented 3 years ago

For further reference - (i know this is tailored for Suricata, but it could be helpfull) https://github.com/pfelk/pfelk/wiki/How-To:-Suricata-on-pfSense

in step 4A - The host-override parameter may be required if Syslog-NG doesn't send the host as the source IP. This overrides the host value for the events when coming into logstash.

I have no issues with suricata ingestion when sending those logs with pfSense as the hostname; but overriding it for the rest of the firewall messages to the IP may be needed?

revere521 commented 3 years ago

You can also do a dump on your pfelk box with sudo tcpdump -vvv -A -i any port 5040 but substitute your actual port (5140?) to see what the actual log messages being sent look like

colinator19 commented 3 years ago

I found that if I disable rename => { "host" => "[observer][ip]" } in the firewall filter within Logstash, I get the below error message which appears to show the actual IP of the firewall and shows the object 'gateway' (coincidence?)

[WARN ] 2021-03-08 23:10:57.513 [[main]>worker0] grok - Grok regexp threw exception {:exception=>"Could not set field 'name' on object 'gateway' to value '192.168.1.1'.This is probably due to trying to set a field like [foo][bar] = someValuewhen [foo] is not either a map or a string", :backtrace=>["org.logstash.Accessors.setChild(Accessors.java:142)", "org.logstash.Accessors.set(Accessors.java:36)", "org.logstash.Event.setField(Event.java:208)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:121)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:84)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:299)", "org.jruby.RubyProc$INVOKER$i$call.call(RubyProc$INVOKER$i$call.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneOrTwoOrNBlock.call(JavaMethod.java:371)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:396)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:205)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:291)", "org.jruby.RubyProc$INVOKER$i$call.call(RubyProc$INVOKER$i$call.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:386)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:184)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:338)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:156)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:143)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:386)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:184)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:191)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:337)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:166)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:108)", "org.jruby.runtime.Block.yield(Block.java:184)", "org.jruby.RubyArray.each(RubyArray.java:1809)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:93)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:546)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:166)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:108)", "org.jruby.runtime.Block.yield(Block.java:184)", "org.jruby.RubyHash$11.visit(RubyHash.java:1434)", "org.jruby.RubyHash$11.visit(RubyHash.java:1431)", "org.jruby.RubyHash.visitLimited(RubyHash.java:698)", "org.jruby.RubyHash.visitAll(RubyHash.java:683)", "org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1391)", "org.jruby.RubyHash.each_pairCommon(RubyHash.java:1426)", "org.jruby.RubyHash.each(RubyHash.java:1415)", "org.jruby.RubyHash$INVOKER$i$0$0$each.call(RubyHash$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:93)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:546)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:156)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:143)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:386)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:184)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:386)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:184)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:191)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther3:do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:106)", "org.jruby.runtime.Block.yield(Block.java:184)", "org.jruby.RubyArray.each(RubyArray.java:1809)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:151)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:160)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther5:each(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", "org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:127)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:134)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset7.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset4.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset3.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset5.compute(Unknown Source)", "org.logstash.generated.CompiledDataset2.compute(Unknown Source)", "org.logstash.generated.CompiledDataset6.compute(Unknown Source)", "org.logstash.generated.CompiledDataset8.compute(Unknown Source)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:329)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:323)", "org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)", "java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)", "java.base/java.lang.reflect.Method.invoke(Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)", "org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:354)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:143)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"], :class=>"Java::OrgLogstash::Accessors::InvalidFieldSetException"}

I did however managed to get the filter.log working now; I had to adjust these three things below to make it work:

  1. Add a rewrite entry in syslog-ng | { set("192.168.8.1", value("HOST")); }; |
  2. Adjust the log entry in syslog-ng to use the rewrite rule | { source(_DEFAULT); rewrite(r_host); destination(syslog-logstash); }; |

From here the observer.ip is still shown as gateway however the IP adres is now being shown in each log record.

  1. I've added this replace => { "[observer][ip]" => "%{[host][name]}" } to the app filter for the firewall in Logstash config.

This might not be the correct solution as I would prefer to have the hostname in the log record (to use as a selector based by name instead of IP) but it does seem to work for now. I will look into this later this week to get some more information.

a3ilson commented 3 years ago

Thanks @colinator19 & @revere521

a3ilson commented 3 years ago

@colinator19 - I would recommend adjusting the item 3 above to: replace => { "[observer][ip]" => "%{[observer][hostname]}" }

I am interested to see how this parses out within Kibana. Thank you for working this too - I can add a guide to the Wiki for others who desire to utilize syslog-ng + the added benefits (e.g. encryption) but will need your assistance or your are certainly welcome to submit a PR.

colinator19 commented 3 years ago

Please note the below is targetted to pfSense 2.5 - not sure if this works on pfSense versions below 2.5 and/or OPNsense. I am hoping other can try the below and do some more tests as well, once it is good and working fine for all logs files it might be usefull to add to the wiki in case someone wants to use the log files directly with syslog-ng on pfSense

Took me a while to get used to working with grok filters but I've got my filterlog working now by reading the logfile directly from syslog-ng (so having syslog on pfsense disabled). I also managed to get both the hostname and IP values correctly as well but it does require a rewrite rule in syslog-ng on the firewall.

Need to do more testing but wanted to share my findings;

Source rule: { wildcard-file( base-dir("/var/log/") filename-pattern("filter.log") recursive(yes) follow-freq(1) program-override("filterlog") flags(no-parse) ); };

Log rule: { source(filterlog); rewrite(r_host); destination(logstash); };

Rewrite rule: { set("192.168.1.1", value("HOST")); };

Example log entry: <13>Mar 13 11:45:17 192.168.1.1 filterlog: Mar 13 11:45:17 vFW01 filterlog[25715]: 6,,,1000000105,mlxen1,match,block,in,6,0x00,0xc0c00,255,UDP,17,238,fe80::8cc:33e9:a1fa:4749,fe02::fb,5353,5353,238

I had to seperate haproxy/suricata filters from the firewall one and apply a different grok filter for the firewall. The firewall filter will look like this: if [observer][type] == "firewall" { grok { match => {"message" => "\<%{POSINT:[log][syslog][priority]}\>?(%{SYSLOGTIMESTAMP:[syslog][created]}|%{TIMESTAMP_ISO8601:[syslog][created]})\s?%{SYSLOGHOST:[host][ip]}\s?%{PROG:[syslog][processname]}\:\s?(%{SYSLOGTIMESTAMP:[event][created]})\s?%{SYSLOGHOST:[host][name]}\s?%{PROG:[process][name]}?\[%{POSINT:[process][pid]}\]\:\s?%{GREEDYDATA:filter_message}"} } date { match => [ "[event][created]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] target => "[event][created]" } }

Full filter will be:

filter { if [observer][type] == "haproxy" or [observer][type] == "suricata" { grok { match => {"message" => "%{POSINT:[log][syslog][priority]}?(%{INT:[log][syslog][version]}\s*)?(%{SYSLOGTIMESTAMP:[event][created]}|%{TIMESTAMP_ISO8601:[event][created]})\s(%{SYSLOGHOST:[host][name]}\s+)?%{PROG:[process][name]}\s*?(\[)?%{POSINT:[process][pid]}(\]:)?\s*(\-\s*\-)?\s*%{GREEDYDATA:filter_message}|%{POSINT:[log][syslog][priority]}?(%{INT:[log][syslog][version]}\s*)?(%{SYSLOGTIMESTAMP:[event][created]}|%{TIMESTAMP_ISO8601:[event][created]})\s(%{SYSLOGHOST:[host][name]}\s+)?%{PROG:[process][name]}\:\s%{GREEDYDATA:filter_message}"} } date { match => [ "[event][created]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] target => "[event][created]" } } if [observer][type] == "firewall" { grok { match => {"message" => "\<%{POSINT:[log][syslog][priority]}\>?(%{SYSLOGTIMESTAMP:[syslog][created]}|%{TIMESTAMP_ISO8601:[syslog][created]})\s?%{SYSLOGHOST:[host][ip]}\s?%{PROG:[syslog][processname]}\:\s?(%{SYSLOGTIMESTAMP:[event][created]})\s?%{SYSLOGHOST:[host][name]}\s?%{PROG:[process][name]}?\[%{POSINT:[process][pid]}\]\:\s?%{GREEDYDATA:filter_message}"} } date { match => [ "[event][created]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] target => "[event][created]" } } }

The app filter should be like this, it includes the IP and hostname as well: ### filterlog ### if [process][name] =~ /^filterlog$/ { mutate { add_tag => [ "firewall" ] add_field => { "[ecs][version]" => "1.7.0" } add_field => { "[event][dataset]" => "pfelk.firewall" } add_field => { "[observer][hostname]" => "%{[host][name]}" } replace => { "[observer][ip]" => "%{[host][ip]}" } } grok { patterns_dir => [ "/etc/pfelk/patterns" ] match => [ "filter_message", "%{PF_LOG_ENTRY}" ] } }

All together, I am getting results in Elastic: image

a3ilson commented 3 years ago

Thanks for the update!

a3ilson commented 3 years ago

@colinator19 - how's the testing going on this endeavor? At some point, I would like to update the wiki but will await your findings.

a3ilson commented 2 years ago

observer.ip filed remove from latest update. Dashboard now filter/pivot based on host.name field.