Bro logs not put in index

[GoogleCodeExporter]

I have been trying all day to get my bro flatfiles (I rsync them over from 
another server into /media/btrfs/bro/*.log) but I cannot get them into my elsa 
node.

What steps will reproduce the problem?
1. Add the syslog config from the documentation on bro integration to your 
syslog config
2. Adjust paths to bro log files as needed
3. Copy new log files into the locations syslog-ng monitors
4. Open elsa web instance, search for bro classes

What is the expected output? What do you see instead?
I would expect to see the contents of say, http.log when I run a search for 
class=BRO_HTTP, but instead I get no results returned.

If I manually edit the syslog-ng.conf file and replace d_elsa's normal perl 
script logging with a file("/path/to/file"), my bro logs properly get written 
to the file. I think something is wrong with how elsa is ingesting the bro logs.

What version of the product are you using? On what operating system?
Latest elsa from install script on CentOS 6.4

Please provide any additional information below.

Original issue reported on code.google.com by i...@pingas.org on 8 Apr 2013 at 2:11

[GoogleCodeExporter]

Do you have "log_parse_errors" enabled in /etc/elsa_node.conf?  If so, do you 
see any parsing errors in /data/elsa/log/node.log?  Can you paste a single line 
successfully written to /path/to/file?

Original comment by mchol...@gmail.com on 8 Apr 2013 at 5:35

[GoogleCodeExporter]

There are a lot of debug and trace statements, but errors are relatively 
infrequent. I get lines like this:
ERROR [2013/04/08 16:11:03] /usr/local/elsa/web/../node//Indexer.pm (2646) 
Indexer::record_host_stats 10707 No latest index found

I'm not sure what to make of it.
Here's an example of what gets sent to d_elsa and does not show up as an index 
(instead intercepted and written to a file):

1365451387      127.0.0.1       bro_dns 14      
1365448732.890931|0vUCZszTho4|192.168.114.129|49157|192.168.114.1|53|udp|55979|t
eredo.ipv6.microsoft.com|1|C_INTERNET|1|A|0|NOERROR|F|F|T|T|0|64.4.25.86,64.4.25
.80,64.4.25.82,64.4.25.84|2122.000000,2122.000000,2122.000000,2122.000000|F     
192.168.114.129 49157   192.168.114.1   53      udp             
teredo.ipv6.microsoft.com       F

Original comment by i...@pingas.org on 8 Apr 2013 at 8:15

[GoogleCodeExporter]

Ok, the logs are making it there and are parsed correctly, so this is a 
database loading issue.  Are there any messages saying "Loaded x logs" or any 
errors about not being able to load logs?

Original comment by mchol...@gmail.com on 8 Apr 2013 at 8:30

[GoogleCodeExporter]

There are no errors about not being able to load logs. AFAIK, logs are never 
put into the MySQL databases. I'm looking at syslog_data and its tables are 
empty.

A search for the word 'Loaded' returns nothing.
Is it normal that the syslog_data tables be blank? Should they have something 
in them?

There aren't many more errors that aren't almost exactly like the previous one 
I mentioned, but here's one:
* ERROR [2013/04/08 15:53:02] /usr/local/elsa/web/../node//Indexer.pm (800) 
Indexer::_check_consolidate 5708 min_id 1 is greater than max_id 0 for table 
sysl
og_data.syslogs_index_1

Original comment by i...@pingas.org on 8 Apr 2013 at 8:37

[GoogleCodeExporter]

This looks like a configuration problem or a MySQL problem.

Please shut down syslog-ng and run this:

cd /usr/local/elsa/node
echo "" | perl elsa.pl -o

This will validate the directory.  Please report the output.

Original comment by mchol...@gmail.com on 9 Apr 2013 at 8:10

[GoogleCodeExporter]

Sorry for the super-late reply.

This is what I get:
Validating directory...
Use of uninitialized value in string eq at /usr/local/elsa/node/Indexer.pm line 
879.
Use of uninitialized value $db_size in concatenation (.) or string at 
/usr/local/elsa/node/Indexer.pm line 198.
Use of uninitialized value $db_size in addition (+) at 
/usr/local/elsa/node/Indexer.pm line 219.
Running once
Use of uninitialized value $line[3] in string eq at 
/usr/local/elsa/node/Reader.pm line 338, <STDIN> line 1.

Original comment by i...@pingas.org on 17 Apr 2013 at 8:39

[GoogleCodeExporter]

Ok, if you startup syslog-ng, does it work now?  If not, look for the latest 
ERROR in in node.log.

Original comment by mchol...@gmail.com on 17 Apr 2013 at 8:51

[GoogleCodeExporter]

Strange... It's starting to insert data now.

I have no idea why it started working again. This is some strange voodoo..

Thank you for your help! Hopefully this continues to work...

Original comment by i...@pingas.org on 17 Apr 2013 at 9:30

[GoogleCodeExporter]

No voodoo, the command I had you run executes the initial_validate_directory 
command which cleans up anything out of place in the database.  Glad to hear 
it's working!

Original comment by mchol...@gmail.com on 18 Apr 2013 at 1:57

• Changed state: Done