Inconsistent syslog stream entries into the Elsa logger MySQL database.

[GoogleCodeExporter]

What steps will reproduce the problem?

1. Configure local iptables to log to local syslog on host to collect logs from
2. Forward those logs via syslog to Elsa log server
3. Validate that this much is working (tcpdump, etc.)

What is the expected output? What do you see instead?

Expectation is to have those syslog streams processed by syslog-ng/elsa.pl, 
inserted into MySQL, indexed by Sphinx, and searchable in Elsa web interface. 
In this example, I'm using iptables as search criteria since it seems to be 
most problematic.

Instead, only sporadic entries from random hosts make it into the MySQL 
database through Syslog/elsa.pl.

What version of the product are you using? On what operating system?

Obtained from install.sh (this is the info you need?)

Eventlog Version 0.2.12
Syslog version 3.2.4

OS is RHEL Server 6.2

Please provide any additional information below.

I have two elsa web servers and two elsa log servers. Rsyslog is used around 
the network to forward all activity from those servers to the loggers. Those 
loggers are indeed seeing specifically the iptables log hits from the network, 
but only 3 of many hosts have their logs recorded in the database, etc. 

I need help debugging the middleware so to speak. Lots of references to files 
in other issues, such as update-from-svn.sh, the syslog-ng.log file, and the 
commands you guys use to debug this are either not present or not apparent. 

Many thanks in advance!

Original issue reported on code.google.com by w.d.clay...@gmail.com on 12 Apr 2012 at 7:31

[GoogleCodeExporter]

Are the two ELSA web/log servers operating independently?  Otherwise, the 
normal setup would be to have a single web server which acts a frontend for 
both log servers.

Is rsyslog forwarding general logs from other devices to ELSA?

"Host" is recorded as the IP address that sent the log, so you will need to 
enable IP spoofing in rsyslog to get the hosts to show up properly.  Would logs 
you expect coming from a given host having a different IP address as host 
account for the difference in what you're expecting to see?

Original comment by mchol...@gmail.com on 12 Apr 2012 at 7:51

[GoogleCodeExporter]

Each ELSA web server is looking at its own ELSA log server independently, if 
that answers the question. In other words they are configured as independent 
stacks. 

All hosts have a *.* syslog forwarder for both log servers though. The data 
sets available to each elsa web instance are 99% identical.

> Are the two ELSA web/log servers operating independently?

I am running rsyslog on the log servers and have those forwarders set up the 
same as every other host - I have verified that syslog-ng is in fact able to 
listen on both TCP and UDP ports and there is not a contention for that socket 
with the local rsyslog instances (lsof, etc).

> Otherwise, the normal setup would be to have a single web server which acts a 
frontend for both log servers.

The stacks are independent with the exception that all devices forward logs to 
both loggers. The chain splits up at the device.

> Is rsyslog forwarding general logs from other devices to ELSA?

Rsyslog is forwarding all logs from other devices to ELSA loggers. I can see 
lots of stuff from, for example, the mail servers and DNS servers, but none of 
which are IPTables log entries. Intentionally violating an IPTables rule fires 
off a syslog message to the loggers, which is verified to be received using 
TCPDump - its the post processing that seems to be problematic.

Where can I get the update-from-svn.sh script?

> "Host" is recorded as the IP address that sent the log, so you will need to 
enable IP spoofing in rsyslog to get the hosts to show up properly.

Searching by "host=192.168.1.X", where X is any number, yields valid results, 
so I think that's working for other log entries. That query just doesn't show 
any results for iptables log entries because they don't even make it into the 
logger's database for some reason.

> Would logs you expect coming from a given host having a different IP address 
as host account for the difference in what you're expecting to see?

No. The network is designed in such a way that this would be extremely 
unlikely. The log entries for iptables that I do see are exactly what I'd 
expect to see from the reported host, as validated by looking at that hosts 
local firewall log.

Original comment by w.d.clay...@gmail.com on 12 Apr 2012 at 8:57

[GoogleCodeExporter]

This may be of some use:

# ./syslog-ng-ctl stats
SourceName;SourceId;SourceInstance;State;Type;Number
source;s_network;;a;processed;18185
destination;d_elsa;;a;processed;18185
global;payload_reallocs;;a;processed;408
global;msg_clones;;a;processed;0
dst.program;d_elsa#0;perl /usr/local/elsa/node/elsa.pl -c 
/etc/elsa_node.conf;a;dropped;0
dst.program;d_elsa#0;perl /usr/local/elsa/node/elsa.pl -c 
/etc/elsa_node.conf;a;processed;18185
dst.program;d_elsa#0;perl /usr/local/elsa/node/elsa.pl -c 
/etc/elsa_node.conf;a;stored;0
center;;queued;a;processed;0
global;sdata_updates;;a;processed;0
center;;received;a;processed;0

Original comment by w.d.clay...@gmail.com on 12 Apr 2012 at 9:02

[GoogleCodeExporter]

update-from-svn.sh is deprecated.  Use the install.sh script in the contrib 
directory with the update command like this:
sh install.sh node update
or
sh install.sh web update
This should only be necessary if you installed some time ago.

What does rsyslog show as received for the iptables log entries?  If I 
understand correctly, you are running ELSA in parallel with the existing 
rsyslog setup, correct?

Original comment by mchol...@gmail.com on 12 Apr 2012 at 10:47

[GoogleCodeExporter]

Rsyslog is not being used to receive logs, if I understand your question 
correctly. I am using the syslog-ng instance that compiles with the install.sh 
script, and it's configuration is unmodified.

Original comment by w.d.clay...@gmail.com on 13 Apr 2012 at 4:58

[GoogleCodeExporter]

Ok, can you have rsyslog (which I am now reading to be the forwarder only) to 
copy iptables logs to a test file locally as well as forward them on to ELSA?  
Or, if you can paste in an anonymized tcpdump entry or two showing what the 
logs look like as they arrive on the ELSA box, that would be helpful as well.  
Thus far, it's sounding like those logs have a weird format, so I want to see 
what they look like.

Original comment by mchol...@gmail.com on 13 Apr 2012 at 5:34

[GoogleCodeExporter]

I'm thinking the log's format is part of the issue as well. Rsyslog is 
currently writing to both local disk and to the wire.

How do you want the packet captures? I don't know how to sanitize a full pcap 
format.

Original comment by w.d.clay...@gmail.com on 13 Apr 2012 at 8:42

[GoogleCodeExporter]

Just a single example log is fine from tcpdump using the -X hexadecimal output 
as text.

Original comment by mchol...@gmail.com on 13 Apr 2012 at 8:47

[GoogleCodeExporter]

Any luck getting this capture?

Original comment by mchol...@gmail.com on 30 Apr 2012 at 2:10

[GoogleCodeExporter]

Closing until I hear back.

Original comment by mchol...@gmail.com on 31 May 2012 at 2:45

• Changed state: Done

[GoogleCodeExporter]

Apologies.

The root cause of this issue may have been due to how log consolidation and
file rotation works. I'm not getting a lot of logs right now, probably less
than 200 messages per second, but during peak utilization hours we'll see a
exponential amount.

Can you recommend some settings for elsa_node.conf for smaller networks
that would be capable of producing massive amounts of logs?

On Thu, May 31, 2012 at 9:45 AM, <
enterprise-log-search-and-archive@googlecode.com> wrote:

Original comment by w.d.clay...@gmail.com on 31 May 2012 at 8:08

[GoogleCodeExporter]

Increase num_indexes to 400, increase allowed_temp_percent to 80.  This should 
give you .8 * 400 * 60 seconds of time before temp indexes get rolled into a 
perm index, and should give you more perm indexes before they get rolled.  If 
that's still not enough, move index_interval up from 60 seconds to something 
larger (this will extend the "lifetime" of a temp index).

Original comment by mchol...@gmail.com on 31 May 2012 at 8:29