beave
commented
5 years ago Thanks for pointing out the health issue. In the future, please make an issue on github.com so it doesn't fall of my radar.By default, Meer only records records (flow, http, dns, etc) around an alert. That is, Meer only looks for "event_type" of "alert". It then extracts the data (json) of that alert. This mimics the functionality of Barnyard2 . IMO storing all flow data, dns, etc. is outside the scope of Meer but is something I am willing to look at. That data would probably be better served going into elasticsearch. The volume of that data would likely over whelm MySQL.I want to add an elasticsearch plugin at some point. Let me know if you have any other questions.On Nov 24, 2018 6:18 PM, tsikerdekis notifications@github.com wrote:I was getting this error after setting up everything for the first time: Successfully connected to MySQL/MariaDB database. [E] [11/24/2018 12:34:36] - [output-plugins/mysql.c, line 106] MySQL/MariaDB Error [1364:] "Field 'health' doesn't have a default value" Offending SQL statement: INSERT INTO sensor (hostname, interface, filter, detail, encoding, last_cid) VALUES ('mysensor:wlp7s0', 'wlp7s0', NULL, '1', '0', '0')
I modified the db to have NULL default value for the health field. Everything works fine now but nothing seems to be captured into mysql: [] [11/24/2018 15:05:33] - @@@@@@@@@@ @@@@@@@@ @@@@@@@@ @@@@@@@ Meer version 0.0.3-git [] [11/24/2018 15:05:33] - @@! @@! @@! @@! @@! @@! @@@ Quadrant Information Security [] [11/24/2018 15:05:33] - @!! !!@ @!@ @!!!:! @!!!:! @!@!!@a https://quadrantsec.com [] [11/24/2018 15:05:33] - !!: !!: !!: !!: !!: :!a Copyright (C) 2018 [] [11/24/2018 15:05:33] - : : : :: :: : :: :: : : : [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - Dropping privileges! [UID: 0 GID: 0] [] [11/24/2018 15:05:33] - Classifications file loaded [/etc/suricata/classification.config]. [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - Decode 'metadata': enabled [] [11/24/2018 15:05:33] - Decode 'flow' : enabled [] [11/24/2018 15:05:33] - Decode 'http' : enabled [] [11/24/2018 15:05:33] - Decode 'tls' : enabled [] [11/24/2018 15:05:33] - Decode 'ssh' : enabled [] [11/24/2018 15:05:33] - Decode 'smtp' : enabled [] [11/24/2018 15:05:33] - Decode 'email' : enabled [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - Waldo loaded. Current position: 4166 [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - --[ SQL information ]-------------------------------------------- [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - SQL Driver: MySQL/MariaDB [] [11/24/2018 15:05:33] - Extra data: enabled [] [11/24/2018 15:05:33] - Legacy Reference System': disabled [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - Successfully connected to MySQL/MariaDB database. [D] [11/24/2018 15:05:33] - SQL Debug: "SELECT sid FROM sensor WHERE hostname='mysensor:wlp7s0' AND interface='wlp7s0' AND detail=1 AND encoding='0'" [] [11/24/2018 15:05:33] - Using Database Sensor ID: 1 [D] [11/24/2018 15:05:33] - SQL Debug: "SELECT last_cid FROM sensor WHERE sid=1 " [] [11/24/2018 15:05:33] - Last CID: 16 [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - Record 'metadata': enabled [] [11/24/2018 15:05:33] - Record 'flow' : enabled [] [11/24/2018 15:05:33] - Record 'http' : enabled [] [11/24/2018 15:05:33] - Record 'tls' : enabled [] [11/24/2018 15:05:33] - Record 'ssh' : enabled [] [11/24/2018 15:05:33] - Record 'smtp' : enabled [] [11/24/2018 15:05:33] - Record 'email' : enabled [] [11/24/2018 15:05:33] - [] [11/24/2018 15:05:33] - --------------------------------------------------------------------------- [] [11/24/2018 15:05:33] - Skipping to record 4166 in /var/log/suricata/eve.json [] [11/24/2018 15:05:33] - Reached target record of 4166. Processing new records. [] [11/24/2018 15:05:33] - Read in 4177 lines [] [11/24/2018 15:05:33] - Waiting for new data...... [D] [11/24/2018 15:06:46] - SQL Debug: "UPDATE sensor SET last_cid='18' WHERE sid=1 AND hostname='mysensor:wlp7s0' AND interface='wlp7s0' AND detail=1" [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - --[ Meer Statistics ]--------------------------------------- [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - - Decoded Statistics: [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - Waldo Postion : 4202 [] [11/24/2018 15:06:47] - Flow : 0 [] [11/24/2018 15:06:47] - HTTP : 0 [] [11/24/2018 15:06:47] - TLS : 0 [] [11/24/2018 15:06:47] - SSH : 0 [] [11/24/2018 15:06:47] - SMTP : 0 [] [11/24/2018 15:06:47] - Email : 0 [] [11/24/2018 15:06:47] - Metadata : 0 [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - - DNS Statistics: [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - DNS Lookups : 0 [] [11/24/2018 15:06:47] - DNS Cache Hits: 0 (0.000%) [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - - MySQL/MariaDB Statistics: [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - Health Checks : 0 [] [11/24/2018 15:06:47] - INSERT : 0 [] [11/24/2018 15:06:47] - SELECT : 2 [] [11/24/2018 15:06:47] - UPDATE : 1 [] [11/24/2018 15:06:47] - Class Cache Misses : 0 [] [11/24/2018 15:06:47] - Class Cache Hits : 0 (0.000%) [] [11/24/2018 15:06:47] - Signature Cache Misses : 0 [] [11/24/2018 15:06:47] - Signature Cache Hits : 0 (0.000%) [] [11/24/2018 15:06:47] - [] [11/24/2018 15:06:47] - Last CID is : 18. [] [11/24/2018 15:06:47] - Shutdown complete.
I know for a fact that eve.json contains all sorts of records and new ones are constantly added to the file. Suricata version I use: Installed: 4.1.0-0ubuntu2 (from the official ppa) Example of a flow record: {"timestamp":"2018-11-24T15:18:00.000281-0800","flow_id":417835289965255,"event_type":"flow","src_ip":"192.168.0.16","src_port":49030,"dest_ip":"239.255.255.250","dest_port":1900,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":3,"pkts_toclient":0,"bytes_toserver":564,"bytes_toclient":0,"start":"2018-11-24T15:17:28.422599-0800","end":"2018-11-24T15:17:28.627439-0800","age":0,"state":"new","reason":"timeout","alerted":false}}
—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.
I was getting this error after setting up everything for the first time:
I modified the db to have NULL default value for the health field. Everything works fine now but nothing seems to be captured into mysql:
I know for a fact that eve.json contains all sorts of records and new ones are constantly added to the file.
Suricata version I use: Installed: 4.1.0-0ubuntu2 (from the official ppa)
Example of a flow record:
{"timestamp":"2018-11-24T15:18:00.000281-0800","flow_id":417835289965255,"event_type":"flow","src_ip":"192.168.0.16","src_port":49030,"dest_ip":"239.255.255.250","dest_port":1900,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":3,"pkts_toclient":0,"bytes_toserver":564,"bytes_toclient":0,"start":"2018-11-24T15:17:28.422599-0800","end":"2018-11-24T15:17:28.627439-0800","age":0,"state":"new","reason":"timeout","alerted":false}}