Some Windows eventids dropped, others logged succesfully -- why?

[GoogleCodeExporter]

In short, I am using Snare (similar to eventlog-to-syslog) to send all my 
domain server's events to ELSA. I see some events (like logon and logoff) but I 
cannot see a lot of important events that I know are getting forwarded (like 
4722 -- user account enabled).

What steps will reproduce the problem?
1. Install snare with most verbose output possible
2. Configure snare to send to elsa
3. Check elsa for certain eventids (I can only see what's listed in the 
attached PNG)

What is the expected output? What do you see instead?
I've been logging a duplicate of everything syslog gets to /root/elsa.debug . 
If I run, say 'cat elsa.debug| grep "|4722|"' I will see this:
1368466854  192.168.14.2    Security    4   May 13 13:40:53 
2013|4722|Microsoft-Windows-Security-Auditing|REDACTED\jdoe|N/A|Success 
Audit|DC1.REDACTED.com|User Account Management||A user account was enabled.    
Subject:   Security ID:  S-1-5-21-2354954538-3863477605-808390733-1664   
Account Name:  iaronson   Account Domain:  REDACTED   Logon ID:  0xe9e969    
Target Account:   Security ID:  S-1-5-21-2354954538-3863477605-808390733-1692   
Account Name:  jdoe   Account Domain:  REDACTED|764360  4722

Which indicates that syslog, at least, has seen the event. I don't understand 
why ELSA isn't seeing it as well. A query of 'eventid:4722' returns no 
results...

What version of the product are you using? On what operating system?
Latest svn revision (I think 878) with up-to-date CentOS 6 for both node and 
web.

Original issue reported on code.google.com by i...@pingas.org on 13 May 2013 at 5:55

Attachments:

• download.png

[GoogleCodeExporter]

If you search for "4722" class:windows, do you get hits on these logs?  How 
about class:none?

Original comment by mchol...@gmail.com on 13 May 2013 at 6:39

[GoogleCodeExporter]

I am not sure what's happening, but in the past 20 minutes I have managed to 
capture two new 4722 events. (I've been generating them on our domain 
controller periodically). However, a third event that I generated 10 minutes 
ago was not picked up (but is still in the syslog).

"4722" class:windows brings up the two events I've been able to see. "4722" 
class:none brings up no events.

It seems that something after syslog is dropping logs when it shouldn't be. I'm 
currently using real time logging with a batch_limit of 10 (I get barely 20 
logs/second max, ever). Should I upload my sphinx and elsa_node configs for you?

Original comment by i...@pingas.org on 13 May 2013 at 6:45

[GoogleCodeExporter]

This sounds like a possible realtime bug, but I can't be sure.  Can you 
determine which events get written to the test file but don't get written to 
ELSA?

Original comment by mchol...@gmail.com on 14 May 2013 at 2:58

[GoogleCodeExporter]

Hmm... I'm not sure what happened now. I left the office for the day and came 
back to try and compare the contents of my debug file against what elsa had 
logged, and now I find that all 5 of my events with eventid:4722 show up in a 
query. After looking at other events, I can't find any that aren't showing up 
in elsa now.

Is there any sort of chance that logs would be delayed showing up in queries 
for an hour or two? Or perhaps a whole day?
This whole problem seems really hard to pin down.

Original comment by i...@pingas.org on 15 May 2013 at 2:53

[GoogleCodeExporter]

I can't imagine how that would happen in realtime mode, but realtime mode
isn't really tested right now (I plan on making it the primary mode of
operation in the next few months).  The only thing I can think of is that
even your batch setting of 10 wasn't enough, but that doesn't really make
sense either.  In realtime, the logs should be available immediately after
the batch limit is hit as it's a direct INSERT into the Sphinx and MySQL
databases.

On Wed, May 15, 2013 at 9:53 AM, <
enterprise-log-search-and-archive@googlecode.com> wrote:

Original comment by mchol...@gmail.com on 15 May 2013 at 3:06

[GoogleCodeExporter]

How strange...

If there's any log files you'd like, I will upload them. Otherwise, I guess you 
should mark this as closed for now.

Original comment by i...@pingas.org on 15 May 2013 at 3:07

[GoogleCodeExporter]

Ok, will close for now.  Please reopen if you find events go missing 
indefinitely.

Original comment by mchol...@gmail.com on 15 May 2013 at 3:09

• Changed state: Done