ELSA not recognizing patterns and not showing the logs on the browser

[GoogleCodeExporter]

Hello! I just recently am involved with ELSA and trying to undestand how it 
works. 
I was able to create some patterns for DHCP logs and all went well and the logs 
show up correctly in the browser. I created some other patterns and this one's 
are not showing up correctly in the browser, the classes exists but as far as I 
know they are not beeing parsed at all. I am storing the patterns in the 
"merged.xml" file as well as "patterndb.xml".
And all of a sudden i am no more receing logs for ELSA. The logs are coming in 
as I can see in tcpdump and going in the DB but just not indexing and showing 
in the browser.
Please can anyone help me with this problem? I would love to fix the problem 
with my patterns and the logs not showing in the browser. :)
Thank you so much :)

Original issue reported on code.google.com by raquel.b...@gmail.com on 23 Jul 2013 at 2:01

[GoogleCodeExporter]

You should place any custom patterns you create in a file (or any number of 
separate files) in /etc/elsa/patterns.d.  merged.xml is overwritten during 
install.sh updates by summarizing all files in /etc/elsa/patterns.d.

Can you try that and let me know if that fixes it?

Original comment by mchol...@gmail.com on 24 Jul 2013 at 6:40

[GoogleCodeExporter]

Hi! thank you for your reply! :)
I just discovered that today and tried it and it works if I put my patterns 
there :) (I didn’t know that I should place the patterns separate, I was 
writing directly into the merged.xml) 

But I continue to have the same problem :/  The merged.xml file is overwritten 
and all is well but when I go to see the logs they don’t appear with the 
classes I created and so. Maybe because I am having trouble indexing the logs I 
think. It seems that ELSA is receiving the logs but not showing them in the 
browser, only if I force the indexing they appear some time after. I use this 
command :
/usr/local/sphinx/bin/indexer --config /usr/local/etc/sphinx.conf --rotate --all
and after a short period of time they appear in the browser. But if I try to 
search for some logs it doesn’t go as well cause the dates may be wrong or 
confused. And as this goes on even if I create the new patterns they are never 
recognized.

Thank you so much for your attention :)

Original comment by raquel.b...@gmail.com on 24 Jul 2013 at 8:33

[GoogleCodeExporter]

It may take up to two minutes for the logs to appear in the browser.  Can you 
confirm that the cron.pl script is running each minute?  The dates on the logs 
should be the timestamp that the log was received off the network by syslog, 
unless you're importing logs.  Is that the case?

Original comment by mchol...@gmail.com on 25 Jul 2013 at 5:39

[GoogleCodeExporter]

I ran again the installation process with your new updated file. When I was 
installing the node part it gave me errors in the databases. And in the web 
part of the installation it gave the following errors:
Executing set_cron
Adding cron entry for alerts...
grep: /var/spool/cron/root: No such file or directory
install.sh: line 1246: /var/spool/cron/root: No such file or directory
set_cron FAIL

So I think cron is not running each minute.
I don't really now why I am getting this much errors :/ And I am not importing 
logs. I receive logs via syslog from a server directly to this machine.

thank you for your time.

Original comment by raquel.b...@gmail.com on 26 Jul 2013 at 10:54

[GoogleCodeExporter]

You may need to locate your cron directory and set it manually.  What OS are 
you on?  You might be able to find it with this command:

find / -name cron | grep root

Then, you need to create a file /etc/elsa_vars.sh and place this in there:

CRONTAB_DIR=(whatever directory you found, minus the "/root" part)

After that, run sh install.sh node update and sh install.sh web update and you 
shouldn't get the error anymore.

Original comment by mchol...@gmail.com on 26 Jul 2013 at 7:43

[GoogleCodeExporter]

thanks! I am going to try that :)

Oh and  am using CentOS.

Original comment by raquel.b...@gmail.com on 29 Jul 2013 at 9:30

[GoogleCodeExporter]

It didn't work with the find command but using locate it gives me this:

[root@elsa ~]# locate cron
/etc/cron.daily
/etc/cron.daily/logrotate
/etc/cron.daily/mlocate.cron
/lib/modules/2.6.32-279.el6.x86_64/kernel/drivers/ata/pata_jmicron.ko
/lib/modules/2.6.32-358.11.1.el6.x86_64/kernel/drivers/ata/pata_jmicron.ko
/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/ata/pata_jmicron.ko
/usr/local/elsa/web/cron.pl
/usr/share/doc/audit-2.2/auditd.cron
/usr/share/man/man8/crond_selinux.8.gz
/usr/share/man/man8/crontab_selinux.8.gz
/usr/share/man/man8/prelink_cron_system_selinux.8.gz
/usr/share/selinux/devel/include/services/cron.if
/var/log/cron
[root@elsa ~]#

Original comment by raquel.b...@gmail.com on 29 Jul 2013 at 9:42

[GoogleCodeExporter]

Ok, we may need to create one.  Try entering "sudo crontab -e"  then paste in 
this:

* * * * * perl /usr/local/elsa/web/cron.pl -c /etc/elsa_web.conf > /dev/null 
2>&1

Save, and exit.  Now, try your find command again and it should be there.

Original comment by mchol...@gmail.com on 29 Jul 2013 at 2:32

[GoogleCodeExporter]

I did what you said in the last comment :)
When I try the locate command it gives me the same as above. 

Is the cron directory this one?

/etc/cron.d ?

[root@elsa cron.d]# ls
0hourly

I can't seem to know what the cron directory is :/

Original comment by raquel.b...@gmail.com on 30 Jul 2013 at 2:26

[GoogleCodeExporter]

well I seem to have resolved the problem. 
My dates were wrong and I have now installed ntp and ajusted the ntp servers to 
be as close to the date in real time.

I now receive the logs in the browser and cron is running. But I didn't need to 
put the cron directory into /etc/elsa_vars.sh

But can you help me with something odd?
I created some patterns for some logs I encountered. For DHCP, Firewall, VPN, 
NAT, etc. The first one (dhcp) works just fine. It appears correctly on the 
browser with the tags I defined and Class=DHCP. But the others don't appear. 
The classes exists, and the patterns are correct but they are not recognized by 
ELSA.
Can you help me with this problem? If you want I can attach the patterns file.
Thanks you for your time.

Original comment by raquel.b...@gmail.com on 30 Jul 2013 at 4:30

[GoogleCodeExporter]

Can you post the patterns you're working on with examples of logs that don't 
match to the mailing list?

Original comment by mchol...@gmail.com on 31 Jul 2013 at 9:07

• Changed state: Done
• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

Here are the patterns I created. The examples are the actual logs that I 
receive. 
The odd thing is when I run this command to test the patterns it gives me no 
errors for the pattern I created.
/usr/local/syslog-ng/bin/pdbtool test -p /usr/local/elsa/node/conf/merged.xml

As well as if I test the logs individually. For example: 
/usr/local/syslog-ng/bin/pdbtool match -p /usr/local/elsa/node/conf/merged.xml 
-P fad -M "Jul 07 11:21:25 WEDT: %ASA-session-6-106015: Deny TCP (no 
connection) from 188.82.84.193/59346 to 193.137.231.238/443 flags FIN ACK on 
interface outside"

it doesn't give out any wrong matches but it just doesn't appear in the 
browser. the only pattern I created that does appear is the DHCP one, the 
others don't and I don't know why.

Original comment by raquel.b...@gmail.com on 1 Aug 2013 at 11:06

Attachments:

• local_patterndb.xml

[GoogleCodeExporter]

for the patterns named "dynamic nat" and "static nat" I created new classes and 
fields for those classes. For the patterns named "vpn" and "fad" I used classes 
already available in the database. for vpn class I didn't add anything just 
created a couple patterns, and for the fad (firewall access deny) I just added 
some fields to the database for my patterns to work. 

I put everything in a file, tested and got no wrong matches but only the dhcp 
patterns work.
I don't know what could be wrong :/

Original comment by raquel.b...@gmail.com on 5 Aug 2013 at 12:04

[GoogleCodeExporter]

Your patterns for nat and vpn have <pattern>nat</pattern> and 
<pattern>vpn</pattern> in them, which should only be the case if the program 
name in the logs was "nat" or "vpn," which I don't think it is.  Try your 
patterns again with removing those <pattern> elements.  Your tests succeeded 
because you declared that "nat" and "vpn" were the program names in the test 
messages, but those aren't the names when the live logs come in.

Original comment by mchol...@gmail.com on 15 Aug 2013 at 1:53