Web interface no longers starts

[GoogleCodeExporter]

What is the expected output? What do you see instead?
I would expect to see the web interface up and running, but instead httpd dies 
(the subsystem stays locked) and I get this error in the log:
[Thu Sep 26 15:35:33 2013] [notice] suEXEC mechanism enabled (wrapper: 
/usr/sbin/suexec)
[Thu Sep 26 15:35:33 2013] [notice] Digest: generating secret for digest 
authentication ...
[Thu Sep 26 15:35:33 2013] [notice] Digest: done
Use of uninitialized value $query in concatenation (.) or string at 
/usr/local/elsa/web/lib/Utils.pm line 137.
[Thu Sep 26 15:35:43 2013] [error] Error while loading 
/usr/local/elsa/web/lib/Web.psgi: Internal error at 
/usr/local/elsa/web/lib/Utils.pm line 143.\nBEGIN failed--compilation aborted 
at /etc/httpd/conf/elsa_startup.pl line 19.\nCompilation failed in require at 
(eval 2) line 1.\n
[Thu Sep 26 15:35:43 2013] [error] Can't load Perl file: 
/etc/httpd/conf/elsa_startup.pl for server elsa.helmpoint.com:0, exiting...

What version of the product are you using? On what operating system?
I am using the latest svn revision on CentOS 6.4

Original issue reported on code.google.com by i...@pingas.org on 26 Sep 2013 at 7:41

[GoogleCodeExporter]

That might be a MySQL connection error.  Is the MySQL server up and running?

Original comment by mchol...@gmail.com on 27 Sep 2013 at 3:28

[GoogleCodeExporter]

After some work, I realized that the VM was out of disk space so I enlarged it.

After enlarging, the disk quickly filled up again... The culprit seems to be a 
file called "syslogs_archive_9431243.ARN" in /var/lib/mysql/syslog_data

Now, I have a separate mount at /data (with 1.4TB free space) that houses the 
bulk of the MySQL tables. Why this extremely large file is not linked to 
/data/elsa/mysql I have no idea...

I assume the ARN is an archive file for MySQL. Is it safe for me to move this 
file onto the /data dir? The lack of free space seems to be causing a plethora 
of issues (including the apache startup failure).

Original comment by i...@pingas.org on 1 Oct 2013 at 6:46

[GoogleCodeExporter]

More info:
# du syslogs_archive_*.ARN -h
29G syslogs_archive_9431243.ARN

This seems really large... 

Original comment by i...@pingas.org on 1 Oct 2013 at 6:47

[GoogleCodeExporter]

Possibly related, I'm getting errors from a cron script that may have been 
installed by ELSA:

# mail
Heirloom Mail version 12.4 7/29/08.  Type ? for help.
"/var/spool/mail/root": 3 messages 3 new
>N  1 Cron Daemon           Tue Oct  1 14:30  23/896   "Cron <root@elsa> 
/usr/lib64/sa/sa1 1 1"
 N  2 Cron Daemon           Tue Oct  1 14:40  23/896   "Cron <root@elsa> /usr/lib64/sa/sa1 1 1"
 N  3 Cron Daemon           Tue Oct  1 14:50  23/896   "Cron <root@elsa> /usr/lib64/sa/sa1 1 1"
& 1
Message  1:
From root@[hostname]  Tue Oct  1 14:30:02 2013
Return-Path: <root@[hostname]>
Date: Tue, 1 Oct 2013 14:30:02 -0400
From: root@[hostname] (Cron Daemon)
To: root@[hostname]
Subject: Cron <root@elsa> /usr/lib64/sa/sa1 1 1
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Status: R

Cannot open /var/log/sa/sa01: No such file or directory

Original comment by i...@pingas.org on 1 Oct 2013 at 6:53

[GoogleCodeExporter]

You can safely delete that huge file, I have no idea why it would be so large.  
I'm assuming it's corrupted and unsuccessfully being repaired.

That cron job is not created by ELSA.

Original comment by mchol...@gmail.com on 1 Oct 2013 at 9:49

[GoogleCodeExporter]

Will the data in the file get rebuilt from the indexes I currently have, or 
does this result in me losing my entire archive?

Original comment by i...@pingas.org on 3 Oct 2013 at 3:27

[GoogleCodeExporter]

I have removed the file, but now searches in ELSA return no results....

For example, this query when using a "from" time from 5 minutes ago (that has 
in the past found all failed sudos on our network in the past 5 minutes) 
crashes: 
"+'authentication failure' program=sudo"

Results in this error in the interface:
"Warnings: No indexes satisfy field requirements, query did not use an index"

Queries without a from time simply result in no data returned.

Original comment by i...@pingas.org on 3 Oct 2013 at 6:00

[GoogleCodeExporter]

Are there any indexes or archive data now?  What do you get with that search 
and no time parameters, with orderby_dir:desc (reverse by timestamp)?

Original comment by mchol...@gmail.com on 9 Oct 2013 at 4:36

[GoogleCodeExporter]

I gave up and just rebuilt a new ELSA installation on a new VM.

If you want to continue to debug this broken installation, I can.

Original comment by i...@pingas.org on 15 Oct 2013 at 8:54

[GoogleCodeExporter]

No worries, let me know if it happens again.

Original comment by mchol...@gmail.com on 15 Oct 2013 at 9:36

• Changed state: Done
• Added labels: Type-Other
• Removed labels: Type-Defect