mysql Fatal Error: Duplicate key

FalcoGer commented 4 years ago

At some point I get duplicate key errors for barnyard. I'm not sure what caused it. There are multiple possibilities

Using BASE to archive some events (and remove them from the database)
Restart barnyard2 engine either with kill -SIGHUP signal or service barnyard2 restart

The main issue here is that this is deemed a fatal error and the demon stops. At this point systemd tries to restart it, but it then runs into the same failure again until systemd gives up.

It would be nice if duplicate keys would be checked on insert and if this would be handled more gracefully. (i.e. Report error to log, rollback transaction, skip event, carry on)

The error in question is:

Apr 16 18:38:16 pi barnyard2[1427]: FATAL ERROR: database mysql_error: Duplicate entry '29210-1' for key 'sig_reference.PRIMARY'
                                            SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('42366','29210','1');]

Also please fix your indentation in spo_database.c

ParagVadher commented 4 years ago

Might not be relevant to answering this question, but could you please tell me what OS and version you are using?

FalcoGer commented 4 years ago

Linux pi 5.3.0-1022-raspi2 #24-Ubuntu SMP Fri Mar 27 21:32:13 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

Ubuntu Server 19.10 eoan for arm64 on a raspberry pi 4

ParagVadher commented 4 years ago

I thought BASE requires a php 5.5 version to run on, which is not supported for 19.10 eoan. you can only run php 5.6+ on ubuntu 14.04 onward... how did you achieve this?

FalcoGer commented 4 years ago

I got a fork of base that runs with php 7 https://github.com/NathanGibbs3/BASE

ParagVadher commented 4 years ago

Thank You, better late than never.

NathanGibbs3 commented 4 years ago

Yeah, BASE is an awesome front end for SNORT. It definitely has it's share of problems and technical debt. Slowly / carefully working through the code base. Documenting / fixing things as I find them. :smile: Thanks for the shout out @FalcoGer .

In a twitter conversation with @secureideas at the beginning of 2019, I was asked if I was planning on maintaining BASE. Wasn't sure then, as I had only ported it to PHP 7x for my own use. At the time, I really didn't know a lot about PHP or SQL.

However, several years earlier, I heard an interview with Kevin on a podcast. He was sharing about how he had come to develop BASE in the first place. During the podcast he jumped out of the middle of the interview for five minutes to tell his kids goodnight. You can't help but respect that. I remember thinking, "WOW! I want to be like that." To say the least, it had a profound impact on my decisions to get involved with open source development and info-sec research.

What I remember Kevin sharing on that podcast was:

The developer of ACID, from which BASE is derived, had built it and disappeared.
Kevin forked ACID and continued it as BASE.
You don't have to be the best to get involved in open source.
You just need to be willing to make something better than it is.

When Kevin told me last year that:

Honestly I haven't touched the project in years. I gave it to someone else and they disappeared. You are welcome to fork it on git and run with it,

Seriously, I could not say no to the man who inspires me to get involved even though I am definitely not the best out there. :smile:

I told him:

I'll create a github repo, dump the 1.4.5 source into it as a starting point, then go from there. I'll still call it BASE, but be clear that it is a continuation of previous work.

Kevin replied:

good luck

As I told him then:

Thanks, I'll need it. Also, Thank You for forking ACID and building BASE in the first place. :-)

Thank you @secureideas for the inspiration. :smile:

secureideas commented 4 years ago

I am not sure how to respond to this. :). I am glad to see others building cool things! I am not sure I should inspire, but am thrilled you were!

ParagVadher commented 4 years ago

@NathanGibbs3 as a newcomer to the world of open-source i find your contribution extremely helpful and inspiring as well. Would love to contribute to this wholesome community consisting of people like @secureideas... You guys are great!

orangemike commented 4 years ago

"output database: log " or "output database: alert" DONOT "output database: log " and "output database: alert"

[log | alert] - specify log or alert to connect the database plugin to the log or alert facility. In most cases you will likely want to use the log facility.

README.database

FalcoGer commented 3 years ago

"output database: log " or "output database: alert" DONOT "output database: log " and "output database: alert"

[log | alert] - specify log or alert to connect the database plugin to the log or alert facility. In most cases you will likely want to use the log facility.

README.database

I'm using

# unified2 
# Recommended for most installs
# output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types
output unified2: filename snort.u2, limit 128
#, appid_event_types

# Additional configuration for specific types of installs
# output alert_unified2: filename snort.alert, limit 128, nostamp
# output log_unified2: filename snort.log, limit 128, nostamp 

# syslog
output alert_syslog: LOG_AUTH LOG_ALERT

Then using barnyard2 to log into DB

output database: log, mysql, user=snort password=**** dbname=snort host=localhost

firnsy / barnyard2

mysql Fatal Error: Duplicate key #256