search

pfelk / pfelk

pfSense/OPNsense + Elastic Stack
https://pfelk.github.io/pfelk/
Other
1.07k stars 192 forks source link

Fix Script and Manual Installations #469

Closed a3ilson closed 1 year ago

a3ilson commented 1 year ago

Consolidating Issues and validate/fix:

a3ilson commented 1 year ago

@tunavis unable to replicate this issue. Please provide log from error-data.sh and be sure to sanitize the password from within the 50-outputs.pfelk file.

tunavis commented 1 year ago

Got i working, but suricata not parsing logs into my ELK stack. using syslog-ng. See attached:

image image image image image image

tunavis commented 1 year ago

And then one more thing is there a dashboard for OPENVPN?

revere521 commented 1 year ago

@tunavis for my Suricata setup in Syslog-ng in pfsense i had to specify the transport as udp: 

{
   network("xxx.xxx.xxx.xxx"
   port(5140)
   transport(udp)
   );
};

I also added a program override and a host override, i don't remember if thats in the instructions

{
  wildcard-file(
    base-dir("/var/log/suricata")
    filename-pattern("eve.json")
    recursive(yes)
    follow-freq(1)
    program-override("suricata")
    host-override("pfSense.HOME") <--- the .HOME here is a reference to my "domain"
    flags(no-parse)
  );
};

The default port in the General settings of syslog.ng may also need to be 5141

From what i can see the suricata settings are the same for your LAN as mine

a3ilson commented 1 year ago

@revere521 - attach those screenshot updates and I’ll update the wiki instructions.

revere521 commented 1 year ago

Sure thing (sorry I haven't been around)

Here is the destination object image

Here is the log object image

Here is the source object I think its important to note that the host override (which is just the firewall hostname) may not be necessary but i hard-coded it image

a3ilson commented 1 year ago

@revere521 are all the steps still relevant within the wiki for Suricata?

revere521 commented 1 year ago

@a3ilson it does look like everything is still OK - there are really only two differences in my config

  1. In my destination output, I specified UDP rather than TCP which works better for me. My network is all local (the pfelk server is internal on the LAN) and i have no failovers, etc. so there may be differences in the destination config dependent on that part.
  2. I set syslog-ng to listen on port 5141 (which it may not be listening to anything since its pulling a local file) - and i cant remember why; probably so it didn't get in between the regular log output for pfsense and pfelk.
revere521 commented 1 year ago

disregard what i noted for the moment - i was behind current on some of the configs -- i just updated everything and i am also no longer seeing any traffic in discover from suricata. I'm going to look back and see whats different, and see if i can fix it

revere521 commented 1 year ago

Ok, it looks like the suricata logs go into the .ds-logs-pfelk-unknown index, i can see them in discover under logs-*

thats with the settings i noted

tunavis commented 1 year ago

image I can confirm this as well @revere521

a3ilson commented 1 year ago

Interesting...would you be able to provide a couple of suricata logs?

Viewing the logs from Kibana>>Discovery>>unknown and providing the original.event?

revere521 commented 1 year ago

Here is the event original from the discover under logs-* 

<13>Mar 23 12:27:03 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:27:03.529983-0400","flow_id":257805375706687,"in_iface":"re0","event_type":"dns","src_ip":"192.168.1.115","src_port":51838,"dest_ip":"192.168.1.1","dest_port":53,"proto":"UDP","dns":{"type":"query","id":31390,"rrname":"api.thingspeak.com","rrtype":"A","tx_id":0}}
<13>Mar 23 12:27:03 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:27:03.532299-0400","flow_id":257805375706687,"in_iface":"re0","event_type":"dns","src_ip":"192.168.1.115","src_port":51838,"dest_ip":"192.168.1.1","dest_port":53,"proto":"UDP","dns":{"version":2,"type":"answer","id":31390,"flags":"8180","qr":true,"rd":true,"ra":true,"rrname":"api.thingspeak.com","rrtype":"A","rcode":"NOERROR","answers":[{"rrname":"api.thingspeak.com","rrtype":"A","ttl":44,"rdata":"34.199.181.212"},{"rrname":"api.thingspeak.com","rrtype":"A","ttl":44,"rdata":"3.225.61.1"}],"grouped":{"A":["34.199.181.212","3.225.61.1"]}}}
<13>Mar 23 12:27:03 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:27:03.533570-0400","flow_id":257805375706687,"in_iface":"re0","event_type":"dns","src_ip":"192.168.1.115","src_port":51838,"dest_ip":"192.168.1.1","dest_port":53,"proto":"UDP","dns":{"type":"query","id":28192,"rrname":"api.thingspeak.com","rrtype":"AAAA","tx_id":2}}
<13>Mar 23 12:30:58 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:30:57.193247-0400","flow_id":1960607452581045,"in_iface":"re0","event_type":"http","src_ip":"192.168.1.107","src_port":54386,"dest_ip":"192.168.254.254","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"192.168.254.254","url":"/images/dev-video-television-3.png","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0","http_content_type":"image/png","accept":"image/avif,image/webp,*/*","accept_encoding":"gzip, deflate","accept_language":"en-US,en;q=0.5","cache_control":"max-age=0","dnt":"1","connection":"close","content_length":"2269","content_type":"image/png","date":"Thu, 23 Mar 2023 16:30:56 GMT","last_modified":"Fri, 30 Apr 2021 18:21:32 GMT","http_refer":"http://192.168.254.254/cgi-bin/home.ha","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":2269}}
<13>Mar 23 12:30:58 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:30:57.141729-0400","flow_id":724376278108939,"in_iface":"re0","event_type":"http","src_ip":"192.168.1.107","src_port":54382,"dest_ip":"192.168.254.254","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"192.168.254.254","url":"/cgi-bin/home.ha","http_user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0","http_content_type":"text/html","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept_encoding":"gzip, deflate","accept_language":"en-US,en;q=0.5","cache_control":"max-age=0","dnt":"1","connection":"close","content_type":"text/html","http_method":"GET","protocol":"HTTP/1.1","status":200,"length":23930}}
<13>Mar 23 12:30:56 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:30:55.403311-0400","flow_id":1625999435310959,"in_iface":"re0","event_type":"dns","src_ip":"192.168.1.162","src_port":26997,"dest_ip":"192.168.1.1","dest_port":53,"proto":"UDP","dns":{"type":"query","id":4756,"rrname":"tabletcaptiveportal.com","rrtype":"A","tx_id":0}}
<13>Mar 23 12:30:56 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:30:55.830537-0400","flow_id":1773394122970833,"in_iface":"re0","event_type":"http","src_ip":"192.168.1.162","src_port":54502,"dest_ip":"3.215.119.161","dest_port":80,"proto":"TCP","tx_id":0,"http":{"hostname":"tabletcaptiveportal.com","url":"/generate_204","http_user_agent":"Dalvik/2.1.0 (Linux; U; Android 9; KFTRWI Build/PS7327.3326N)","accept_encoding":"gzip","connection":"keep-alive","date":"Thu, 23 Mar 2023 16:30:55 GMT","http_method":"GET","protocol":"HTTP/1.1","status":204,"length":0}}
<13>Mar 23 12:30:56 pfSense.HOME suricata: {"timestamp":"2023-03-23T12:30:55.703071-0400","flow_id":1625999435310959,"in_iface":"re0","event_type":"dns","src_ip":"192.168.1.162","src_port":26997,"dest_ip":"192.168.1.1","dest_port":53,"proto":"UDP","dns":{"version":2,"type":"answer","id":4756,"flags":"8180","qr":true,"rd":true,"ra":true,"rrname":"tabletcaptiveportal.com","rrtype":"A","rcode":"NOERROR","answers":[{"rrname":"tabletcaptiveportal.com","rrtype":"A","ttl":20,"rdata":"3.215.119.161"},{"rrname":"tabletcaptiveportal.com","rrtype":"A","ttl":20,"rdata":"52.201.39.120"},{"rrname":"tabletcaptiveportal.com","rrtype":"A","ttl":20,"rdata":"18.211.145.60"},{"rrname":"tabletcaptiveportal.com","rrtype":"A","ttl":20,"rdata":"34.232.236.215"},{"rrname":"tabletcaptiveportal.com","rrtype":"A","ttl":20,"rdata":"52.6.242.213"},{"rrname":"tabletcaptiveportal.com","rrtype":"A","ttl":20,"rdata":"44.199.173.144"}],"grouped":{"A":["3.215.119.161","52.201.39.120","18.211.145.60","34.232.236.215","52.6.242.213","44.199.173.144"]}}}
a3ilson commented 1 year ago

Thanks! Can you provide a couple of firewall logs too?

The grok is failing and want to confirm the altered format

a3ilson commented 1 year ago

Or update the pfelk.grok, restart logstash. The updated GROK should correct the issue but will also need to confirm the format with pfSense's RFC5424 syslog format...don't think it'll matter with Suricata as those logs are being sent via syslog-ng.

# pfelk.grok
################################################################################
# Version: 23.03f                                                              #
#                                                                              #
#                                                                              #
#                                                                              #
################################################################################
#
#
# PFELK
PFELK (%{PFSENSE}|%{OPNSENSE})

# pfSense
PFSENSE (%{PFSENSE_LOG}|%{PFSENSE5424_LOG})
PFSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\s%{POSINT:[log][syslog][procid]}(\s\-\s\-\s)%{GREEDYDATA:filter_message}
PFSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s%{PROG:[log][syslog][appname]}\[%{POSINT:[log][syslog][procid]}\]\:\s%{GREEDYDATA:filter_message}
PFSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s%{PROG:[log][syslog][appname]}(\[%{POSINT:[log][syslog][procid]}\]\:)?\s%{GREEDYDATA:filter_message}

# OPNsense
OPNSENSE (%{OPNSENSE_LOG}|%{OPNSENSE5424_LOG})
OPNSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\s%{POSINT:[log][syslog][procid]}\s-\s\[meta\ssequenceId\=\"%{NUMBER:[event][sequence]}\"\]\s%{GREEDYDATA:filter_message}
OPNSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\[%{POSINT:[log][syslog][procid]}\]\:\s%{GREEDYDATA:filter_message}

# CAPTIVE PORTAL (Optional)
CAPTIVEPORTAL (%{CP_PFSENSE}|%{CP_OPNSENSE})
CP_OPNSENSE %{WORD:[event][action]}\s%{GREEDYDATA:[client][user][name]}\s\(%{IP:[client][ip]}\)\s%{WORD:[observer][ingress][interface][alias]}\s%{INT:[observer][ingress][zone]}
# ToDo - Clean-up pfSense GROK pattern below
CP_PFSENSE (%{CAPTIVE1}|%{CAPTIVE2})
CAPTIVE1 %{WORD:[observer][ingress][interface][alias]}:\s%{DATA:[observer][ingress][zone]}\s\-\s%{WORD:[event][action]}\:\s%{GREEDYDATA:[client][user][name]},\s%{MAC:[client][mac]},\s%{IP:[client][ip]}(,\s%{GREEDYDATA:[event][reason]})?
CAPTIVE2 %{WORD:[observer][ingress][interface][alias]}:\s%{DATA:[observer][ingress][zone]}\s\-\s%{GREEDYDATA:[event][action]}\:\s%{GREEDYDATA:[client][user][name]},\s%{MAC:[client][mac]},\s%{IP:[client][ip]}(,\s%{GREEDYDATA:[event][reason]})?

# DHCPv4 (Optional)
DHCPD DHCP(%{DHCPD_DISCOVER}|%{DHCPD_DUPLICATE}|%{DHCPD_OFFER_ACK}|%{DHCPD_REQUEST}|%{DHCPD_DECLINE}|%{DHCPD_RELEASE}|%{DHCPD_INFORM}|%{DHCPD_LEASE})|%{DHCPD_REUSE}|%{DHCPDv6}|(%{GREEDYDATA:[DHCPD][message]})?
DHCPD_DISCOVER (?<[dhcp][operation]>DISCOVER) from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_DECLINE (?<[dhcp][operation]>DECLINE) of %{IP:[dhcpv4][client][ip]} from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_DUPLICATE uid %{WORD:[dhcp][operation]} %{IP:[dhcpv4][client][ip]} for client %{MAC:[dhcpv4][client][mac]} is %{WORD:[error][code]} on %{GREEDYDATA:[dhcpv4][client][address]}
DHCPD_INFORM (?<[dhcp][operation]>INFORM) from %{IP:[dhcpv4][client][ip]}? %{DHCPD_VIA}
DHCPD_LEASE (?<[dhcp][operation]>LEASE(QUERY|UNKNOWN|ACTIVE|UNASSIGNED)) (from|to) %{IP:[dhcpv4][client][ip]} for (IP %{IP:[dhcpv4][query][ip]}|client-id %{NOTSPACE:[dhcpv4][query][id]}|MAC address %{MAC:[dhcpv4][query][mac]})( \(%{NUMBER:[dhcpv4][query][associated]} associated IPs\))?
DHCPD_OFFER_ACK (?<[dhcp][operation]>(OFFER|N?ACK)) on %{IP:[dhcpv4][client][ip]} to %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_RELEASE (?<[dhcp][operation]>RELEASE) of %{IP:[dhcpv4][client][ip]} from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA} \((?<dhcpd_release>(not )?found)\)
DHCPD_REQUEST (?<[dhcp][operation]>REQUEST) for %{IP:[dhcpv4][client][ip]}( \(%{DATA:[dhcpv4][server][ip]}\))? from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_VIA via (%{IP:[dhcpv4][relay][ip]}|(?<[interface][name]>[^: ]+))
DHCPD_REUSE (?<[dhcpv4][operation]>reuse_lease): lease age %{INT:[dhcpv4][lease][duration]}.* lease for %{IPV4:[dhcpv4][client][ip]}

# DHCPv6 (Optional - In Development)
DHCPDv6 (%{DHCPv6_REPLY}|%{DHCPv6_ACTION}|%{DHCPv6_REUSE})
DHCPv6_REPLY (?<[dhcpv6][operation]>Advertise|Reply) NA: address %{IP:[dhcpv6][client][ip]} to client with duid %{GREEDYDATA:[dhcpv6][duid]}\siaid\s\=\s%{INT:[dhcpv6][iaid]} valid for %{INT:[dhcpv6][lease][duration]} seconds
DHCPv6_ACTION (?<[dhcpv6][operation]>(Request|Picking|Sending Reply|Sending Advertise|Confirm|Solicit|Renew))(\s)?(message)?(\s)?(to|from)?(\s)?(pool address)? %{IP:[dhcpv6][client][ip]}(\s)?(port %{INT:[dhcpv6][client][port]})?(, transaction ID %{BASE16FLOAT:[dhcpv6][transaction][id]})?
DHCPv6_REUSE (?<[dhcpv6][operation]>Reusing lease) for: %{IPV6:[dhcpv6][client][ip]}, age %{INT:[dhcpv6][lease][age]}.*preferred: %{INT:[dhcpv6][lease][age][preferred]}, valid %{INT:[dhcpv6][lease][age][valid]}

# HAPROXY
HA_PROXY (%{HAPROXY}|%{HAPROXY_TCP})
HAPROXY %{IP:[client][ip]}:%{INT:[client][port]} \[%{HAPROXYDATE:[haproxy][timestamp]}\] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]}/%{INT:[haproxy][time_backend_response]}/%{NOTSPACE:[host][uptime]} %{INT:[http][response][status_code]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][http][request][captured_cookie]} %{DATA:[haproxy][http][response][captured_cookie]} %{NOTSPACE:[haproxy][termination_state]} %{INT:[haproxy][connections][active]}/%{INT:[haproxy][connections][frontend]}/%{INT:[haproxy][connections][backend]}/%{INT:[haproxy][connections][server]}/%{NOTSPACE:[haproxy][connections][retries]} %{INT:[haproxy][server_queue]}/%{INT:[haproxy][backend_queue]} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:[http][request][method]} (%{URIPROTO:[haproxy][mode]}://)?(?:%{USER:[user][name]}(?::[^@]*)?@)?(?:%{URIHOST:[http][request][referrer]})?(?:%{URIPATHPARAM:[http][mode]})?( HTTP/%{NUMBER:[http][version]})?))?"?
HAPROXY_TCP %{IP:[haproxy][client][ip]}:%{INT:[haproxy][client][port]} [%{HAPROXYDATE:haproxy_timestamp}] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]} %{INT:[haproxy][time_backend_response]}%{NOTSPACE:[haproxy][time_duration]}%{INT:[haproxy][http_status_code]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][captured_request_cookie]}%{DATA:[haproxy][captured_response_cookie]} %{NOTSPACE:[haproxy][termination_state]}

# NGINX
NGINX %{NGINX_META}%{NGINX_LOG}(%{NGINX_EXT})?
NGINX_META %{IPORHOST:[client][ip]}(\s\-\s)(%{USERNAME:[nginx][access][user_name]}|\-)?\s\[%{HTTPDATE:timestamp}\]\s*\"
NGINX_LOG %{WORD:[nginx][access][method]}\s*%{NOTSPACE:[nginx][access][url]}\s*HTTP/%{NUMBER:[nginx][access][http_version]}\"\s%{NUMBER:[nginx][access][response_code]}\s%{NUMBER:[nginx][access][body_sent][bytes]}\s"%{NOTSPACE:[nginx][access][referrer]}"\s"%{DATA:[nginx][access][agent]}"
NGINX_EXT (\s\"\-\"\s*)\"%{IPORHOST:[nginx][access][forwarder]}\"(%{NGINX_EXT_SN}%{NGINX_EXT_RT}%{NGINX_EXT_UA}%{NGINX_EXT_US}%{NGINX_EXT_UT}%{NGINX_EXT_UL})?
NGINX_EXT_SN \s*sn=(\"%{HOSTNAME:[nginx][ingress_controller][upstream][name]}\"|"")
NGINX_EXT_RT \s*rt=(%{NUMBER:[nginx][ingress_controller][http][request][time]}|"")
NGINX_EXT_UA \s*ua=(("%{IP:[nginx][ingress_controller][upstream][ip]}:%{INT:[nginx][ingress_controller][upstream][port]}")|("-")|("%{NOTSPACE:[nginx][ingress_controller][upstream][socket]}"))\s*
NGINX_EXT_US \s*us=(\"%{NUMBER:[nginx][ingress_controller][upstream][response][status_code]}\"|"-")
NGINX_EXT_UT \s*ut=(\"%{NUMBER:[nginx][ingress_controller][upstream][response][time]}\"|"-")
NGINX_EXT_UL \s*ul=(\"%{NUMBER:[nginx][ingress_controller][upstream][response][length]}\"|"-")

# OPENVPN - Initial Filter
OPENVPN (%{IP:[openvpn][user][ip]}.%{INT:[openvpn][user][port]})?(%{USERNAME:[openvpn][user]})?(.*\(tos\s*%{BASE16NUM:[openvpn][tos]},\s*ttl\s*%{INT:[openvpn][ttl]},\s*id\s*%{POSINT:[openvpn][process][ppid]},\s*offset\s*%{INT:[openvpn][offset]},\s*flags\s*\[%{WORD:[openvpn][flags]}\],\s*proto\s*%{WORD:[openvpn][protocol][type]}\s*\(%{INT:[openvpn][protocol][id]}\),\s*length\s*%{NUMBER:[openvpn][packet][length]}\)\s*%{IP:[openvpn][client][ip]}\.%{INT:[openvpn][client][port]}\s*>\s*%{IP:[openvpn][server][ip]}\.%{INT:[openvpn][server][port]}:\s*\[%{GREEDYDATA:[openvpn][checksum]}\]\s*%{WORD:[openvpn][network][transport]},\s*length\s*%{INT:[openvpn][transport][data_length]})?

# PF
PF_LOG_ENTRY %{PF_LOG_DATA}%{PF_IP_SPECIFIC_DATA}%{PF_IP_DATA}%{PF_PROTOCOL_DATA}?
PF_LOG_DATA %{INT:[rule][ruleset]},%{INT:[rule][id]}?,,%{DATA:[rule][uuid]},%{DATA:[interface][name]},(?<[event][reason]>\b[\w\-]+\b),%{WORD:[event][action]},%{WORD:[network][direction]},
PF_IP_SPECIFIC_DATA %{PF_IPv4_SPECIFIC_DATA}|%{PF_IPv6_SPECIFIC_DATA}
PF_IPv4_SPECIFIC_DATA (?<[network][type]>(4)),%{BASE16NUM:[pf][ipv4][tos]},%{WORD:[pf][ipv4][ecn]}?,%{INT:[pf][ipv4][ttl]},%{INT:[pf][ipv4][packet][id]},%{INT:[pf][ipv4][offset]},%{WORD:[pf][ipv4][flags]},%{INT:[network][iana_number]},%{WORD:[network][transport]},
PF_IP_DATA %{INT:[pf][packet][length]},%{IP:[source][ip]},%{IP:[destination][ip]},
PF_PROTOCOL_DATA %{PF_TCP_DATA}|%{PF_UDP_DATA}|%{PF_ICMP_DATA}|%{PF_IGMP_DATA}|%{PF_IPv6_VAR}|%{PF_IPv6_ICMP}
PF_IPv6_SPECIFIC_DATA (?<[network][type]>(6)),%{BASE16NUM:[pf][ipv6][class]},%{WORD:[pf][ipv6][flow_label]},%{WORD:[pf][ipv6][hop_limit]},%{DATA:[pf][protocol][type]},%{INT:[pf][protocol][id]},
PF_IPv6_VAR %{WORD:type},%{WORD:option},%{WORD:Flags},%{WORD:Flags}
PF_IPv6_ICMP

# PF PROTOCOL
PF_TCP_DATA %{INT:[source][port]},%{INT:[destination][port]},%{INT:[pf][transport][data_length]},(?<[pf][tcp][flags]>(\w*)?),(?<[pf][tcp][sequence_number]>(\d*)?):?\d*,(?<[pf][tcp][ack_number]>(\d*)?),(?<[pf][tcp][window]>(\d*)?),(?<[pf][tcp][urg]>(\w*)?),%{GREEDYDATA:[pf][tcp][options]}
PF_UDP_DATA %{INT:[source][port]},%{INT:[destination][port]},%{INT:[pf][transport][data_length]}$
PF_IGMP_DATA datalength=%{INT:[network][packets]}
PF_ICMP_DATA %{PF_ICMP_TYPE}%{PF_ICMP_RESPONSE}
PF_ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
PF_ICMP_RESPONSE %{PF_ICMP_ECHO_REQ_REPLY}|%{PF_ICMP_UNREACHPORT}|%{PF_ICMP_UNREACHPROTO}|%{PF_ICMP_UNREACHABLE}|%{PF_ICMP_NEED_FLAG}|%{PF_ICMP_TSTAMP}|%{PF_ICMP_TSTAMP_REPLY}
PF_ICMP_ECHO_REQ_REPLY %{INT:[pf][icmp][echo][id]},%{INT:[pf][icmp][echo][sequence]}
PF_ICMP_UNREACHPORT %{IP:[pf][icmp][unreachport][destination][ip]},%{WORD:[pf][icmp][unreachport][protocol]},%{INT:[pf][icmp][unreachport][port]}
PF_ICMP_UNREACHPROTO %{IP:[pf][icmp][unreach][destination][ip]},%{WORD:[pf][icmp][unreach][network][transport]}
PF_ICMP_UNREACHABLE %{GREEDYDATA:[pf][icmp][unreachable]}
PF_ICMP_NEED_FLAG %{IP:[pf][icmp][need_flag][ip]},%{INT:[pf][icmp][need_flag][mtu]}
PF_ICMP_TSTAMP %{INT:[pf][icmp][tstamp][id]},%{INT:[pf][icmp][tstamp][sequence]}
PF_ICMP_TSTAMP_REPLY %{INT:[pf][icmp][tstamp][reply][id]},%{INT:[pf][icmp][tstamp][reply][sequence]},%{INT:[pf][icmp][tstamp][reply][otime]},%{INT:[pf][icmp][tstamp][reply][rtime]},%{INT:[pf][icmp][tstamp][reply][ttime]}
PF_SPEC \+

# PF
PF_CARP_DATA (%{WORD:[pf][carp][type]}),(%{INT:[pf][carp][ttl]}),(%{INT:[pf][carp][vhid]}),(%{INT:[pf][carp][version]}),(%{INT:[pf][carp][advbase]}),(%{INT:[pf][carp][advskew]})
PF_APP (%{DATA:[pf][app][page]}):
PF_APP_DATA (%{PF_APP_LOGOUT}|%{PF_APP_LOGIN}|%{PF_APP_ERROR}|%{PF_APP_GEN})
PF_APP_LOGIN (%{DATA:[pf][app][action]}) for user \'(%{DATA:[pf][app][user]})\' from: (%{IP:[pf][remote][ip]})
PF_APP_LOGOUT User (%{DATA:[pf][app][action]}) for user \'(%{DATA:[pf][app][user]})\' from: (%{IP:[pf][remote][ip]})
PF_APP_ERROR webConfigurator (%{DATA:[pf][app][action]}) for user \'(%{DATA:[pf][app][user]})\' from (%{IP:[pf][remote][ip]})
PF_APP_GEN (%{GREEDYDATA:[pf][app][action]})

# SURICATA
SURICATA \[%{NUMBER:[suricata][rule][uuid]}:%{NUMBER:[suricata][rule][id]}:%{NUMBER:[suricata][rule][version]}\]%{SPACE}%{GREEDYDATA:[suricata][rule][description]}%{SPACE}\[Classification:%{SPACE}%{GREEDYDATA:[suricata][rule][category]}\]%{SPACE}\[Priority:%{SPACE}%{NUMBER:[suricata][priority]}\]%{SPACE}{%{WORD:[network][transport]}}%{SPACE}%{IP:[source][ip]}:%{NUMBER:[source][port]}%{SPACE}->%{SPACE}%{IP:[destination][ip]}:%{NUMBER:[destination][port]}

# SNORT
SNORT \[%{INT:[rule][uuid]}\:%{INT:[rule][reference]}\:%{INT:[rule][version]}\].%{GREEDYDATA:[vulnerability][description]}.\[Classification\: %{DATA:[vulnerability][classification]}\].\[Priority\: %{INT:[event][severity]}\].\{%{DATA:[network][transport]}\}.%{IP:[source][ip]}(\:%{INT:[source][port]})?.->.%{IP:[destination][ip]}(\:%{INT:[destination][port]})?

# SQUID
SQUID %{IPORHOST:[client][ip]} %{NOTSPACE:[labels][request_status]}/%{NUMBER:[http][response][body][status_code]} %{NUMBER:[http][response][bytes]} %{NOTSPACE:[http][request][method]} (%{URIPROTO:[url][scheme]}://)?(?<[url][domain]>\S+?)(:%{INT:[url][port]})?(/%{NOTSPACE:[url][path]})?\s+%{NOTSPACE:[http][request][referrer]}\s+%{NOTSPACE:[lables][hierarchy_status]}/%{NOTSPACE:[destination][address]}\s+%{NOTSPACE:[http][response][mime_type]}

# UNBOUND
UNBOUND %{INT:[process][pgid]}:%{INT:[process][thread][id]}] %{LOGLEVEL:[log][level]}: %{IP:[client][ip]} %{GREEDYDATA:[dns][question][name]}\. %{WORD:[dns][question][type]} %{WORD:[dns][question][class]}
revere521 commented 1 year ago

I updated the grok, sorry busy afternoon - i'll let it run for a bit and then i can look at any of the log/event original you need

revere521 commented 1 year ago

The new grok doesn't seem to have picked up the suricata items, and there are several items still being placed in logs_*

Here is a big .csv from the discover view of logs-* - column AT is the event original

Untitled discover search.csv

a3ilson commented 1 year ago

Thanks!

try this variant:

# pfelk.grok
################################################################################
# Version: 23.03g                                                              #
#                                                                              #
#                                                                              #
#                                                                              #
################################################################################
#
#
# PFELK
PFELK (%{PFSENSE}|%{OPNSENSE})

# pfSense
PFSENSE (%{PFSENSE_LOG}|%{PFSENSE5424_LOG})
PFSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\s%{POSINT:[log][syslog][procid]}(\s\-\s\-\s)%{GREEDYDATA:filter_message}
PFSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s(%{SYSLOGHOST:[log][syslog][hostname]}\s)?%{PROG:[log][syslog][appname]}(\[%{POSINT:[log][syslog][procid]}\])?\:\s%{GREEDYDATA:filter_message}

# OPNsense
OPNSENSE (%{OPNSENSE_LOG}|%{OPNSENSE5424_LOG})
OPNSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\s%{POSINT:[log][syslog][procid]}\s-\s\[meta\ssequenceId\=\"%{NUMBER:[event][sequence]}\"\]\s%{GREEDYDATA:filter_message}
OPNSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\[%{POSINT:[log][syslog][procid]}\]\:\s%{GREEDYDATA:filter_message}

# CAPTIVE PORTAL (Optional)
CAPTIVEPORTAL (%{CP_PFSENSE}|%{CP_OPNSENSE})
CP_OPNSENSE %{WORD:[event][action]}\s%{GREEDYDATA:[client][user][name]}\s\(%{IP:[client][ip]}\)\s%{WORD:[observer][ingress][interface][alias]}\s%{INT:[observer][ingress][zone]}
# ToDo - Clean-up pfSense GROK pattern below
CP_PFSENSE (%{CAPTIVE1}|%{CAPTIVE2})
CAPTIVE1 %{WORD:[observer][ingress][interface][alias]}:\s%{DATA:[observer][ingress][zone]}\s\-\s%{WORD:[event][action]}\:\s%{GREEDYDATA:[client][user][name]},\s%{MAC:[client][mac]},\s%{IP:[client][ip]}(,\s%{GREEDYDATA:[event][reason]})?
CAPTIVE2 %{WORD:[observer][ingress][interface][alias]}:\s%{DATA:[observer][ingress][zone]}\s\-\s%{GREEDYDATA:[event][action]}\:\s%{GREEDYDATA:[client][user][name]},\s%{MAC:[client][mac]},\s%{IP:[client][ip]}(,\s%{GREEDYDATA:[event][reason]})?

# DHCPv4 (Optional)
DHCPD DHCP(%{DHCPD_DISCOVER}|%{DHCPD_DUPLICATE}|%{DHCPD_OFFER_ACK}|%{DHCPD_REQUEST}|%{DHCPD_DECLINE}|%{DHCPD_RELEASE}|%{DHCPD_INFORM}|%{DHCPD_LEASE})|%{DHCPD_REUSE}|%{DHCPDv6}|(%{GREEDYDATA:[DHCPD][message]})?
DHCPD_DISCOVER (?<[dhcp][operation]>DISCOVER) from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_DECLINE (?<[dhcp][operation]>DECLINE) of %{IP:[dhcpv4][client][ip]} from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_DUPLICATE uid %{WORD:[dhcp][operation]} %{IP:[dhcpv4][client][ip]} for client %{MAC:[dhcpv4][client][mac]} is %{WORD:[error][code]} on %{GREEDYDATA:[dhcpv4][client][address]}
DHCPD_INFORM (?<[dhcp][operation]>INFORM) from %{IP:[dhcpv4][client][ip]}? %{DHCPD_VIA}
DHCPD_LEASE (?<[dhcp][operation]>LEASE(QUERY|UNKNOWN|ACTIVE|UNASSIGNED)) (from|to) %{IP:[dhcpv4][client][ip]} for (IP %{IP:[dhcpv4][query][ip]}|client-id %{NOTSPACE:[dhcpv4][query][id]}|MAC address %{MAC:[dhcpv4][query][mac]})( \(%{NUMBER:[dhcpv4][query][associated]} associated IPs\))?
DHCPD_OFFER_ACK (?<[dhcp][operation]>(OFFER|N?ACK)) on %{IP:[dhcpv4][client][ip]} to %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_RELEASE (?<[dhcp][operation]>RELEASE) of %{IP:[dhcpv4][client][ip]} from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA} \((?<dhcpd_release>(not )?found)\)
DHCPD_REQUEST (?<[dhcp][operation]>REQUEST) for %{IP:[dhcpv4][client][ip]}( \(%{DATA:[dhcpv4][server][ip]}\))? from %{MAC:[dhcpv4][client][mac]}( \(%{DATA:[dhcpv4][option][hostname]}\))? %{DHCPD_VIA}
DHCPD_VIA via (%{IP:[dhcpv4][relay][ip]}|(?<[interface][name]>[^: ]+))
DHCPD_REUSE (?<[dhcpv4][operation]>reuse_lease): lease age %{INT:[dhcpv4][lease][duration]}.* lease for %{IPV4:[dhcpv4][client][ip]}

# DHCPv6 (Optional - In Development)
DHCPDv6 (%{DHCPv6_REPLY}|%{DHCPv6_ACTION}|%{DHCPv6_REUSE})
DHCPv6_REPLY (?<[dhcpv6][operation]>Advertise|Reply) NA: address %{IP:[dhcpv6][client][ip]} to client with duid %{GREEDYDATA:[dhcpv6][duid]}\siaid\s\=\s%{INT:[dhcpv6][iaid]} valid for %{INT:[dhcpv6][lease][duration]} seconds
DHCPv6_ACTION (?<[dhcpv6][operation]>(Request|Picking|Sending Reply|Sending Advertise|Confirm|Solicit|Renew))(\s)?(message)?(\s)?(to|from)?(\s)?(pool address)? %{IP:[dhcpv6][client][ip]}(\s)?(port %{INT:[dhcpv6][client][port]})?(, transaction ID %{BASE16FLOAT:[dhcpv6][transaction][id]})?
DHCPv6_REUSE (?<[dhcpv6][operation]>Reusing lease) for: %{IPV6:[dhcpv6][client][ip]}, age %{INT:[dhcpv6][lease][age]}.*preferred: %{INT:[dhcpv6][lease][age][preferred]}, valid %{INT:[dhcpv6][lease][age][valid]}

# HAPROXY
HA_PROXY (%{HAPROXY}|%{HAPROXY_TCP})
HAPROXY %{IP:[client][ip]}:%{INT:[client][port]} \[%{HAPROXYDATE:[haproxy][timestamp]}\] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]}/%{INT:[haproxy][time_backend_response]}/%{NOTSPACE:[host][uptime]} %{INT:[http][response][status_code]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][http][request][captured_cookie]} %{DATA:[haproxy][http][response][captured_cookie]} %{NOTSPACE:[haproxy][termination_state]} %{INT:[haproxy][connections][active]}/%{INT:[haproxy][connections][frontend]}/%{INT:[haproxy][connections][backend]}/%{INT:[haproxy][connections][server]}/%{NOTSPACE:[haproxy][connections][retries]} %{INT:[haproxy][server_queue]}/%{INT:[haproxy][backend_queue]} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:[http][request][method]} (%{URIPROTO:[haproxy][mode]}://)?(?:%{USER:[user][name]}(?::[^@]*)?@)?(?:%{URIHOST:[http][request][referrer]})?(?:%{URIPATHPARAM:[http][mode]})?( HTTP/%{NUMBER:[http][version]})?))?"?
HAPROXY_TCP %{IP:[haproxy][client][ip]}:%{INT:[haproxy][client][port]} [%{HAPROXYDATE:haproxy_timestamp}] %{NOTSPACE:[haproxy][frontend_name]} %{NOTSPACE:[haproxy][backend_name]}/%{NOTSPACE:[haproxy][server_name]} %{INT:[haproxy][time_request]}/%{INT:[haproxy][time_queue]}/%{INT:[haproxy][time_backend_connect]} %{INT:[haproxy][time_backend_response]}%{NOTSPACE:[haproxy][time_duration]}%{INT:[haproxy][http_status_code]} %{NOTSPACE:[haproxy][bytes_read]} %{DATA:[haproxy][captured_request_cookie]}%{DATA:[haproxy][captured_response_cookie]} %{NOTSPACE:[haproxy][termination_state]}

# NGINX
NGINX %{NGINX_META}%{NGINX_LOG}(%{NGINX_EXT})?
NGINX_META %{IPORHOST:[client][ip]}(\s\-\s)(%{USERNAME:[nginx][access][user_name]}|\-)?\s\[%{HTTPDATE:timestamp}\]\s*\"
NGINX_LOG %{WORD:[nginx][access][method]}\s*%{NOTSPACE:[nginx][access][url]}\s*HTTP/%{NUMBER:[nginx][access][http_version]}\"\s%{NUMBER:[nginx][access][response_code]}\s%{NUMBER:[nginx][access][body_sent][bytes]}\s"%{NOTSPACE:[nginx][access][referrer]}"\s"%{DATA:[nginx][access][agent]}"
NGINX_EXT (\s\"\-\"\s*)\"%{IPORHOST:[nginx][access][forwarder]}\"(%{NGINX_EXT_SN}%{NGINX_EXT_RT}%{NGINX_EXT_UA}%{NGINX_EXT_US}%{NGINX_EXT_UT}%{NGINX_EXT_UL})?
NGINX_EXT_SN \s*sn=(\"%{HOSTNAME:[nginx][ingress_controller][upstream][name]}\"|"")
NGINX_EXT_RT \s*rt=(%{NUMBER:[nginx][ingress_controller][http][request][time]}|"")
NGINX_EXT_UA \s*ua=(("%{IP:[nginx][ingress_controller][upstream][ip]}:%{INT:[nginx][ingress_controller][upstream][port]}")|("-")|("%{NOTSPACE:[nginx][ingress_controller][upstream][socket]}"))\s*
NGINX_EXT_US \s*us=(\"%{NUMBER:[nginx][ingress_controller][upstream][response][status_code]}\"|"-")
NGINX_EXT_UT \s*ut=(\"%{NUMBER:[nginx][ingress_controller][upstream][response][time]}\"|"-")
NGINX_EXT_UL \s*ul=(\"%{NUMBER:[nginx][ingress_controller][upstream][response][length]}\"|"-")

# OPENVPN - Initial Filter
OPENVPN (%{IP:[openvpn][user][ip]}.%{INT:[openvpn][user][port]})?(%{USERNAME:[openvpn][user]})?(.*\(tos\s*%{BASE16NUM:[openvpn][tos]},\s*ttl\s*%{INT:[openvpn][ttl]},\s*id\s*%{POSINT:[openvpn][process][ppid]},\s*offset\s*%{INT:[openvpn][offset]},\s*flags\s*\[%{WORD:[openvpn][flags]}\],\s*proto\s*%{WORD:[openvpn][protocol][type]}\s*\(%{INT:[openvpn][protocol][id]}\),\s*length\s*%{NUMBER:[openvpn][packet][length]}\)\s*%{IP:[openvpn][client][ip]}\.%{INT:[openvpn][client][port]}\s*>\s*%{IP:[openvpn][server][ip]}\.%{INT:[openvpn][server][port]}:\s*\[%{GREEDYDATA:[openvpn][checksum]}\]\s*%{WORD:[openvpn][network][transport]},\s*length\s*%{INT:[openvpn][transport][data_length]})?

# PF
PF_LOG_ENTRY %{PF_LOG_DATA}%{PF_IP_SPECIFIC_DATA}%{PF_IP_DATA}%{PF_PROTOCOL_DATA}?
PF_LOG_DATA %{INT:[rule][ruleset]},%{INT:[rule][id]}?,,%{DATA:[rule][uuid]},%{DATA:[interface][name]},(?<[event][reason]>\b[\w\-]+\b),%{WORD:[event][action]},%{WORD:[network][direction]},
PF_IP_SPECIFIC_DATA %{PF_IPv4_SPECIFIC_DATA}|%{PF_IPv6_SPECIFIC_DATA}
PF_IPv4_SPECIFIC_DATA (?<[network][type]>(4)),%{BASE16NUM:[pf][ipv4][tos]},%{WORD:[pf][ipv4][ecn]}?,%{INT:[pf][ipv4][ttl]},%{INT:[pf][ipv4][packet][id]},%{INT:[pf][ipv4][offset]},%{WORD:[pf][ipv4][flags]},%{INT:[network][iana_number]},%{WORD:[network][transport]},
PF_IP_DATA %{INT:[pf][packet][length]},%{IP:[source][ip]},%{IP:[destination][ip]},
PF_PROTOCOL_DATA %{PF_TCP_DATA}|%{PF_UDP_DATA}|%{PF_ICMP_DATA}|%{PF_IGMP_DATA}|%{PF_IPv6_VAR}|%{PF_IPv6_ICMP}
PF_IPv6_SPECIFIC_DATA (?<[network][type]>(6)),%{BASE16NUM:[pf][ipv6][class]},%{WORD:[pf][ipv6][flow_label]},%{WORD:[pf][ipv6][hop_limit]},%{DATA:[pf][protocol][type]},%{INT:[pf][protocol][id]},
PF_IPv6_VAR %{WORD:type},%{WORD:option},%{WORD:Flags},%{WORD:Flags}
PF_IPv6_ICMP

# PF PROTOCOL
PF_TCP_DATA %{INT:[source][port]},%{INT:[destination][port]},%{INT:[pf][transport][data_length]},(?<[pf][tcp][flags]>(\w*)?),(?<[pf][tcp][sequence_number]>(\d*)?):?\d*,(?<[pf][tcp][ack_number]>(\d*)?),(?<[pf][tcp][window]>(\d*)?),(?<[pf][tcp][urg]>(\w*)?),%{GREEDYDATA:[pf][tcp][options]}
PF_UDP_DATA %{INT:[source][port]},%{INT:[destination][port]},%{INT:[pf][transport][data_length]}$
PF_IGMP_DATA datalength=%{INT:[network][packets]}
PF_ICMP_DATA %{PF_ICMP_TYPE}%{PF_ICMP_RESPONSE}
PF_ICMP_TYPE (?<icmp_type>(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),
PF_ICMP_RESPONSE %{PF_ICMP_ECHO_REQ_REPLY}|%{PF_ICMP_UNREACHPORT}|%{PF_ICMP_UNREACHPROTO}|%{PF_ICMP_UNREACHABLE}|%{PF_ICMP_NEED_FLAG}|%{PF_ICMP_TSTAMP}|%{PF_ICMP_TSTAMP_REPLY}
PF_ICMP_ECHO_REQ_REPLY %{INT:[pf][icmp][echo][id]},%{INT:[pf][icmp][echo][sequence]}
PF_ICMP_UNREACHPORT %{IP:[pf][icmp][unreachport][destination][ip]},%{WORD:[pf][icmp][unreachport][protocol]},%{INT:[pf][icmp][unreachport][port]}
PF_ICMP_UNREACHPROTO %{IP:[pf][icmp][unreach][destination][ip]},%{WORD:[pf][icmp][unreach][network][transport]}
PF_ICMP_UNREACHABLE %{GREEDYDATA:[pf][icmp][unreachable]}
PF_ICMP_NEED_FLAG %{IP:[pf][icmp][need_flag][ip]},%{INT:[pf][icmp][need_flag][mtu]}
PF_ICMP_TSTAMP %{INT:[pf][icmp][tstamp][id]},%{INT:[pf][icmp][tstamp][sequence]}
PF_ICMP_TSTAMP_REPLY %{INT:[pf][icmp][tstamp][reply][id]},%{INT:[pf][icmp][tstamp][reply][sequence]},%{INT:[pf][icmp][tstamp][reply][otime]},%{INT:[pf][icmp][tstamp][reply][rtime]},%{INT:[pf][icmp][tstamp][reply][ttime]}
PF_SPEC \+

# PF
PF_CARP_DATA (%{WORD:[pf][carp][type]}),(%{INT:[pf][carp][ttl]}),(%{INT:[pf][carp][vhid]}),(%{INT:[pf][carp][version]}),(%{INT:[pf][carp][advbase]}),(%{INT:[pf][carp][advskew]})
PF_APP (%{DATA:[pf][app][page]}):
PF_APP_DATA (%{PF_APP_LOGOUT}|%{PF_APP_LOGIN}|%{PF_APP_ERROR}|%{PF_APP_GEN})
PF_APP_LOGIN (%{DATA:[pf][app][action]}) for user \'(%{DATA:[pf][app][user]})\' from: (%{IP:[pf][remote][ip]})
PF_APP_LOGOUT User (%{DATA:[pf][app][action]}) for user \'(%{DATA:[pf][app][user]})\' from: (%{IP:[pf][remote][ip]})
PF_APP_ERROR webConfigurator (%{DATA:[pf][app][action]}) for user \'(%{DATA:[pf][app][user]})\' from (%{IP:[pf][remote][ip]})
PF_APP_GEN (%{GREEDYDATA:[pf][app][action]})

# SURICATA
SURICATA \[%{NUMBER:[suricata][rule][uuid]}:%{NUMBER:[suricata][rule][id]}:%{NUMBER:[suricata][rule][version]}\]%{SPACE}%{GREEDYDATA:[suricata][rule][description]}%{SPACE}\[Classification:%{SPACE}%{GREEDYDATA:[suricata][rule][category]}\]%{SPACE}\[Priority:%{SPACE}%{NUMBER:[suricata][priority]}\]%{SPACE}{%{WORD:[network][transport]}}%{SPACE}%{IP:[source][ip]}:%{NUMBER:[source][port]}%{SPACE}->%{SPACE}%{IP:[destination][ip]}:%{NUMBER:[destination][port]}

# SNORT
SNORT \[%{INT:[rule][uuid]}\:%{INT:[rule][reference]}\:%{INT:[rule][version]}\].%{GREEDYDATA:[vulnerability][description]}.\[Classification\: %{DATA:[vulnerability][classification]}\].\[Priority\: %{INT:[event][severity]}\].\{%{DATA:[network][transport]}\}.%{IP:[source][ip]}(\:%{INT:[source][port]})?.->.%{IP:[destination][ip]}(\:%{INT:[destination][port]})?

# SQUID
SQUID %{IPORHOST:[client][ip]} %{NOTSPACE:[labels][request_status]}/%{NUMBER:[http][response][body][status_code]} %{NUMBER:[http][response][bytes]} %{NOTSPACE:[http][request][method]} (%{URIPROTO:[url][scheme]}://)?(?<[url][domain]>\S+?)(:%{INT:[url][port]})?(/%{NOTSPACE:[url][path]})?\s+%{NOTSPACE:[http][request][referrer]}\s+%{NOTSPACE:[lables][hierarchy_status]}/%{NOTSPACE:[destination][address]}\s+%{NOTSPACE:[http][response][mime_type]}

# UNBOUND
UNBOUND %{INT:[process][pgid]}:%{INT:[process][thread][id]}] %{LOGLEVEL:[log][level]}: %{IP:[client][ip]} %{GREEDYDATA:[dns][question][name]}\. %{WORD:[dns][question][type]} %{WORD:[dns][question][class]}
revere521 commented 1 year ago

that seems to have done the trick with suricata, the index is populating, i can see events in discover and in the dashboard. I still see some stuff going into logs-* but it might be the pfsense rule matching stuff. I'm going to let it run for a bit and see what happens. I can post another csv if you want to take a look

a3ilson commented 1 year ago

Excellent!

Those odds and ends should be captured within the unknown-*

If you don't mind providing another export...I can take a look and see what other logs can be enriched/parsed.

revere521 commented 1 year ago

i don't mind at all, here you go

Untitled discover search 2.csv

a3ilson commented 1 year ago

Thanks! is that an export from discover>>logs-*? It appears most are parsing correctly.

I fixed the ngnix issue and those should parse correctly now.

The only outliers are dhcpclient, sshguard, dhcpclient and usr\sbin\cron which should be good as unknown-*

revere521 commented 1 year ago

Yes sir, its all that remains in the Logs-* discover after the last grok variant. I'm good with that! thanks

revere521 commented 1 year ago

Also referring back to my syslog-ng setup and screenshots, that is still the same and working for me.

YannisHeine commented 1 year ago

Hey, just found this issue - seems related to my problem. I just setup the syslog-ng.

The suricata logs from syslog-ng end up in the logs-pfelk-unknown index.

details:

image

a3ilson commented 1 year ago

@YannisHeine have you updated to the latest version? This issue was corrected/fixed.

tunavis commented 1 year ago

All looks to be working now thanks @a3ilson

2 Things:

  1. I will my destination syslog-ng look like if I use a certificate?

  2. Do we have a dashboard for current OPENVPN Users?

tunavis commented 1 year ago

Sorry and then i seen no events are showing in my Discover table::

image

YannisHeine commented 1 year ago

@a3ilson yea, i freshly installed

YannisHeine commented 1 year ago

@a3ilson ah i guess i see what happend i used the https://github.com/pfelk/docker repo there is the fix not present i think

a3ilson commented 1 year ago

Correct… I haven’t updated the docker repo yet

tunavis commented 1 year ago

Hi Guys can someone send me a example of the syslog-ng using certs I have the following certs that needs to be imported somehow in my pfsense: ca.crt, logstash.crt, logstash.pkcs8.key

image

And the one more thing is there a dashboard for Openvpn?