Can't see any queries...

GoogleCodeExporter commented 8 years ago

What steps will reproduce the problem?
Not sure where exactly to start... All of a sudden one day, our web frontend 
just started turning up nothing. There appears to be no data in the pshinx 
indexes. Can you help me figure out why my data isn't showing up anymore?

We have a node instance on one machine that is fed bro, snort, and windows 
event logs via syslog on the local machine. We also have a web frontend that 
queries that node over the LAN to get results.

What version of the product are you using? On what operating system?
Node -- Archlinux, not sure if latest svn
Web -- CentOS 6.2, updating web frontend now.

Please provide any additional information below.

Original issue reported on code.google.com by i...@pingas.org on 7 Dec 2012 at 3:58

GoogleCodeExporter commented 8 years ago

Ok, first question: by "nothing" do you mean that the page loads but there are 
no results for any searches?

If the page loads and there are no results, the first thing I would want to see 
is the output from a SQL query on the node like this:
mysql syslog_data -e "select * from v_directory; select * from tables;"

Original comment by mchol...@gmail.com on 7 Dec 2012 at 4:10

GoogleCodeExporter commented 8 years ago

Well, previously I got an error along the lines of "Node returned empty result 
for query!". I just ran an update on the centos web install, and now I'm 
getting a 500 internal server error. I've rebooted the centos box to see if 
that might help. I can attach the apache error log if that will help (once the 
box comes back online).

Attached is the output of the mysql command. There is no v_directory table, it 
seems. That can't be good...

Original comment by i...@pingas.org on 7 Dec 2012 at 4:26

Attachments:

output.txt

GoogleCodeExporter commented 8 years ago

Attached is the httpd error log on the web node. I'm still getting the 500 
error and the default apache error page for it.

Original comment by i...@pingas.org on 7 Dec 2012 at 4:28

Attachments:

error_log

GoogleCodeExporter commented 8 years ago

install.sh should have installed the Perl module URI::Encode, which the error 
says is not installed.  Please try manually installing with:

sudo cpanm URI::Encode

My mistake on the SQL command: it should be run on syslog, not syslog_data, so 
please run:

mysql syslog -e "select * from v_directory; select * from tables;"

Original comment by mchol...@gmail.com on 7 Dec 2012 at 4:35

GoogleCodeExporter commented 8 years ago

Weird, both cpan and cpanm (as well as yaml) weren't installed on the centos 
machine. Not sure how that happened. They're all installed now, as is 
URI::Encode. Still getting a 500 on the web frontend. Error log attached. Looks 
like it's missing more modules? Should I just re-run the update script?

Here's the output of the sql command, below (Sorry, it's a mess, but that's how 
it came out):

+------+---------------------+---------------------+--------+--------+----------
+---------+-------------------------------+-----------------+---------------+---
---------+----------+------+-----------+-----------------+-------------+--------
---+-----------------+---------------+
| id   | start               | end                 | min_id | max_id | first_id 
| last_id | table_name                    | table_start_int | table_end_int | 
table_type | table_id | type | locked_by | table_locked_by | index_start | 
index_end | index_start_int | index_end_int |
+------+---------------------+---------------------+--------+--------+----------
+---------+-------------------------------+-----------------+---------------+---
---------+----------+------+-----------+-----------------+-------------+--------
---+-----------------+---------------+
| NULL | 2012-10-07 13:28:50 | 2012-12-07 11:54:59 |      1 | 657292 |     NULL 
|    NULL | syslog_data.syslogs_archive_1 |      1349630930 |    1354899299 | 
archive    |        4 | NULL |      NULL |            NULL | NULL        | NULL 
     |            NULL |          NULL |
+------+---------------------+---------------------+--------+--------+----------
+---------+-------------------------------+-----------------+---------------+---
---------+----------+------+-----------+-----------------+-------------+--------
---+-----------------+---------------+
+----+-------------------------------+---------------------+--------------------
-+--------+--------+---------------+-----------------+
| id | table_name                    | start               | end                
 | min_id | max_id | table_type_id | table_locked_by |
+----+-------------------------------+---------------------+--------------------
-+--------+--------+---------------+-----------------+
|  4 | syslog_data.syslogs_archive_1 | 2012-10-07 13:28:50 | 2012-12-07 
11:54:59 |      1 | 657292 |             2 |            NULL |
+----+-------------------------------+---------------------+--------------------
-+--------+--------+---------------+-----------------+

Original comment by i...@pingas.org on 7 Dec 2012 at 4:58

Attachments:

error_log

GoogleCodeExporter commented 8 years ago

Yes, the logs look good.  Just re-run the install.sh web update script and the 
modules should get installed.  Could it have been a VM that was reverted?

Original comment by mchol...@gmail.com on 7 Dec 2012 at 5:00

GoogleCodeExporter commented 8 years ago

[deleted comment]

GoogleCodeExporter commented 8 years ago

Possibly, I'm not the only one in control of our VMs.

I get the error below when I run the ./install.sh web update command. I know it 
says I should ignore any errors, but could it be relevant?

I'm still getting a 500 on the web interface. Attached is the error log.

Exported revision 555.
get_elsa success
Executing update_web_mysql
Updating web MySQL, please ignore any errors for this section...
ERROR 1091 (42000) at line 1: Can't DROP 'action_params'; check that column/key 
exists
Finished updating MySQL
update_web_mysql success
!!!!!! IMPORTANT !!!!!!!!!
If you have a host-based firewall like IPTables running, remember to allow 
ports 80 (and/or 443) for the web server and ports 514 (syslog), 3306 (MySQL), 
and 3307 (Sphinx) for log nodes

Original comment by i...@pingas.org on 7 Dec 2012 at 6:11

Attachments:

error_log

GoogleCodeExporter commented 8 years ago

As per the message, you can ignore the MySQL "ERROR" as it's ok.  I've fixed 
install.sh so it shouldn't be reported anymore.

So, now you're missing the Perl module Data::Google::Visualization::DataSource, 
which you can install with cpanm.  I'm confused as to why that didn't get 
installed via web update.  It doesn't look like it's running based on your 
above output.  Can you make sure you're invoking the install.sh script with "sh 
install.sh web update" instead of "./install.sh?"  I want to make sure we're on 
the same page.

Original comment by mchol...@gmail.com on 7 Dec 2012 at 7:07

GoogleCodeExporter commented 8 years ago

Here's the whole output when I run the script. I had to manually answer yes 
twice during the GeoIP install:

~ # sh install.sh web update                                                    

                                    (12/07 02:13 pm)
Assuming distro to be centos
Executing centos_get_web_packages
Loaded plugins: fastestmirror, refresh-packagekit
Loading mirror speeds from cached hostfile
 * base: centos.mirror.balt.rackd.net
 * extras: mirrors.greenmountainaccess.net
 * updates: mirrors.lga7.us.voxel.net
Setting up Update Process
No Packages marked for Update
Loaded plugins: fastestmirror, refresh-packagekit
Ignored option -q, -v, -d or -e (probably due to merging: -yq != -y -q)
Loading mirror speeds from cached hostfile
 * base: centos.mirror.balt.rackd.net
 * extras: mirrors.greenmountainaccess.net
 * updates: mirrors.lga7.us.voxel.net
Setting up Install Process
Package curl-7.19.7-26.el6_2.4.x86_64 already installed and latest version
Package subversion-1.6.11-7.el6.x86_64 already installed and latest version
Package 1:make-3.81-20.el6.x86_64 already installed and latest version
Package gcc-4.4.6-4.el6.x86_64 already installed and latest version
Package gcc-c++-4.4.6-4.el6.x86_64 already installed and latest version
Package mysql-5.1.66-1.el6_3.x86_64 already installed and latest version
Package mysql-libs-5.1.66-1.el6_3.x86_64 already installed and latest version
Package mysql-devel-5.1.66-1.el6_3.x86_64 already installed and latest version
Package httpd-2.2.15-15.el6.centos.1.x86_64 already installed and latest version
Package mod_perl-2.0.4-10.el6.x86_64 already installed and latest version
Package pam-devel-1.1.1-10.el6_2.1.x86_64 already installed and latest version
Package setools-console-3.3.7-4.el6.x86_64 already installed and latest version
Nothing to do
centos_get_web_packages success
Executing set_date
 7 Dec 14:13:14 ntpdate[3406]: adjust time server 128.138.141.172 offset -0.435718 sec
set_date success
Executing check_svn_proxy
check_svn_proxy success
Executing build_web_perl
Sys::Info is up to date. (0.78)
Authen::PAM is up to date. (0.16)
Time::HiRes is up to date. (1.9725)
Moose is up to date. (2.0604)
Config::JSON is up to date. (1.5100)
Plack::Builder is up to date. (undef)
Date::Manip is up to date. (6.37)
Digest::SHA1 is up to date. (2.13)
MIME::Base64 is up to date. (3.13)
URI::Escape is up to date. (3.31)
Socket is up to date. (2.006)
Net::DNS is up to date. (0.70)
Sys::Hostname::FQDN is up to date. (0.11)
String::CRC32 is up to date. (1.4)
CHI is up to date. (0.56)
Search::QueryParser is up to date. (0.94)
AnyEvent::DBI is up to date. (2.2)
DBD::mysql is up to date. (4.022)
EV is up to date. (4.11)
Sys::Info is up to date. (0.78)
Sys::MemInfo is up to date. (0.91)
MooseX::Traits is up to date. (0.11)
Authen::Simple is up to date. (0.5)
Authen::Simple::PAM is up to date. (0.2)
Authen::Simple::DBI is up to date. (0.2)
Authen::Simple::LDAP is up to date. (0.3)
Net::LDAP::Express is up to date. (0.11)
Net::LDAP::FilterBuilder is up to date. (1.0004)
Plack::Middleware::CrossOrigin is up to date. (0.007)
Module::Pluggable is up to date. (4.5)
Module::Install is up to date. (1.06)
PDF::API2::Simple is up to date. (1.1.4)
XML::Writer is up to date. (0.615)
Parse::Snort is up to date. (0.6)
Spreadsheet::WriteExcel is up to date. (2.38)
IO::String is up to date. (1.08)
Mail::Internet is up to date. (2.11)
Log::Log4perl is up to date. (1.40)
Email::LocalDelivery is up to date. (0.217)
Plack::Session is up to date. (0.15)
CHI::Driver::DBI is up to date. (1.24)
Plack::Builder::Conditionals is up to date. (0.03)
AnyEvent::HTTP is up to date. (2.15)
URL::Encode is up to date. (0.01)
MooseX::ClassAttribute is up to date. (0.26)
Data::Serializable is up to date. (0.40.1)
MooseX::Log::Log4perl is up to date. (0.46)
Plack::Middleware::NoMultipleSlashes is up to date. (0.001)
MooseX::Storage is up to date. (0.32)
MooseX::Clone is up to date. (0.05)
Retrieving GeoIP databases...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.5M  100 11.5M    0     0  4104k      0  0:00:02  0:00:02 --:--:-- 4168k
gzip: /tmp/GeoLiteCity.dat already exists; do you wish to overwrite (y or n)? y
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  471k  100  471k    0     0  1814k      0 --:--:-- --:--:-- --:--:-- 1950k
gzip: /tmp/GeoIP.dat already exists; do you wish to overwrite (y or n)? y
...done.
Using slower pure-Perl GeoIP library, install GeoIP C library for faster version
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  109k  100  109k    0     0   170k      0 --:--:-- --:--:-- --:--:--  568k
Geo-IP-1.40/
Geo-IP-1.40/Changes
Geo-IP-1.40/example/
Geo-IP-1.40/example/benchmark.pl
Geo-IP-1.40/example/city.pl
Geo-IP-1.40/example/country_v6.pl
Geo-IP-1.40/example/geolite-mirror-simple.pl
Geo-IP-1.40/example/netspeed.pl
Geo-IP-1.40/example/netspeedcell.pl
Geo-IP-1.40/example/org.pl
Geo-IP-1.40/example/region.pl
Geo-IP-1.40/INSTALL
Geo-IP-1.40/IP.xs
Geo-IP-1.40/lib/
Geo-IP-1.40/lib/Geo/
Geo-IP-1.40/lib/Geo/IP/
Geo-IP-1.40/lib/Geo/IP/Record.pm
Geo-IP-1.40/lib/Geo/IP/Record.pod
Geo-IP-1.40/lib/Geo/IP.pm
Geo-IP-1.40/lib/Geo/Mirror.pm
Geo-IP-1.40/Makefile.PL
Geo-IP-1.40/MANIFEST
Geo-IP-1.40/META.json
Geo-IP-1.40/META.yml
Geo-IP-1.40/ppport.h
Geo-IP-1.40/README
Geo-IP-1.40/t/
Geo-IP-1.40/t/0_base.t
Geo-IP-1.40/t/1_lookup.t
Geo-IP-1.40/t/20_min_capi_version.t
Geo-IP-1.40/t/2_namelookup.t
Geo-IP-1.40/t/3_mirror.t
Geo-IP-1.40/t/cpan_mirror.txt
Geo-IP-1.40/typemap
Note (probably harmless): No library found for -lGeoIP
Writing Makefile for Geo::IP
Writing MYMETA.yml and MYMETA.json
Skip blib/lib/Geo/Mirror.pm (unchanged)
Skip blib/lib/Geo/IP/Record.pod (unchanged)
Skip blib/lib/Geo/IP/Record.pm (unchanged)
Skip blib/lib/Geo/IP.pm (unchanged)
Manifying blib/man3/Geo::Mirror.3pm
Manifying blib/man3/Geo::IP::Record.3pm
Manifying blib/man3/Geo::IP.3pm
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 
'blib/lib', 'blib/arch')" t/*.t
t/0_base.t ............... ok   
t/1_lookup.t ............. ok     
t/20_min_capi_version.t .. ok   
t/2_namelookup.t ......... ok     
t/3_mirror.t ............. ok   
All tests successful.
Files=5, Tests=34,  1 wallclock secs ( 0.06 usr  0.07 sys +  0.38 cusr  0.29 
csys =  0.80 CPU)
Result: PASS
Manifying blib/man3/Geo::Mirror.3pm
Manifying blib/man3/Geo::IP::Record.3pm
Manifying blib/man3/Geo::IP.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
build_web_perl success
Executing get_elsa
A    elsa
A    elsa/contrib
A    elsa/contrib/install.sh
A    elsa/contrib/forward_novell_logs.pl
A    elsa/contrib/syslog-ng
A    elsa/contrib/searchd.freebsd
A    elsa/contrib/httpry_logger.pl
A    elsa/contrib/symantec
A    elsa/contrib/symantec/INSTALL
A    elsa/contrib/symantec/symantec_logger.pl
A    elsa/contrib/update_from_svn.sh
A    elsa/contrib/monitoring
A    elsa/contrib/monitoring/check_all_required_procs.pl
A    elsa/contrib/monitoring/INSTALL
A    elsa/contrib/searchd
A    elsa/contrib/dashboards
A    elsa/contrib/dashboards/snort.js
A    elsa/contrib/dashboards/bro.js
A    elsa/contrib/sophos
A    elsa/contrib/sophos/patterndb.xml
A    elsa/contrib/sophos/INSTALL
A    elsa/contrib/sophos/schema.sql
A    elsa/contrib/bulk_query.pl
A    elsa/contrib/Ironport
A    elsa/contrib/Ironport/patterndb.xml
A    elsa/contrib/Ironport/INSTALL
A    elsa/contrib/Ironport/patterndb.xml.corr
A    elsa/contrib/Ironport/syslog-ng-corr.conf
A    elsa/contrib/netflow
A    elsa/contrib/netflow/netflow_syslog.pl
A    elsa/LICENSE
A    elsa/node
A    elsa/node/.project
A    elsa/node/conf
A    elsa/node/conf/stopwords.txt
A    elsa/node/conf/patterndb.xml
A    elsa/node/conf/syslog-ng.conf
A    elsa/node/conf/elsa.conf
A    elsa/node/conf/schema.sql
A    elsa/node/Writer.pm
A    elsa/node/Indexer.pm
A    elsa/node/log2timeline2elsa.pl
A    elsa/node/Reader.pm
A    elsa/node/elsa.pl
A    elsa/INSTALL
A    elsa/README
A    elsa/web
A    elsa/web/cli.pl
A    elsa/web/conf
A    elsa/web/conf/meta_db_schema.mysql
A    elsa/web/conf/elsa.conf
A    elsa/web/conf/apache_site.conf
A    elsa/web/conf/example_dashboard.json
A    elsa/web/lib
A    elsa/web/lib/Query.pm
A    elsa/web/lib/Export
A    elsa/web/lib/Export/HTML.pm
A    elsa/web/lib/Export/Spreadsheet.pm
A    elsa/web/lib/Export/PDF.pm
A    elsa/web/lib/Export/CSV.pm
A    elsa/web/lib/Export/KML.pm
A    elsa/web/lib/YUI.pm
A    elsa/web/lib/Fields.pm
A    elsa/web/lib/API.pm
A    elsa/web/lib/Transform.pm
A    elsa/web/lib/Dashboard
A    elsa/web/lib/Dashboard/File.pm
A    elsa/web/lib/Web.pm
A    elsa/web/lib/User.pm
A    elsa/web/lib/Connector
A    elsa/web/lib/Connector/MapReduce.pm
A    elsa/web/lib/Connector/Email.pm
A    elsa/web/lib/Connector/StreamGrep.pm
A    elsa/web/lib/Connector/Sandbox.pm
A    elsa/web/lib/Connector/CIF.pm
A    elsa/web/lib/Connector/SIRT.pm
A    elsa/web/lib/Datasource
A    elsa/web/lib/Datasource/Database.pm
A    elsa/web/lib/Info
A    elsa/web/lib/Info/Snort.pm
A    elsa/web/lib/Info/Windows.pm
A    elsa/web/lib/Info/Pcap.pm
A    elsa/web/lib/Export.pm
A    elsa/web/lib/Dashboard.pm
A    elsa/web/lib/Datasource.pm
A    elsa/web/lib/Connector.pm
A    elsa/web/lib/Info.pm
A    elsa/web/lib/API
A    elsa/web/lib/API/Charts.pm
A    elsa/web/lib/Transform
A    elsa/web/lib/Transform/Interval.pm
A    elsa/web/lib/Transform/Database.pm
A    elsa/web/lib/Transform/Sum.pm
A    elsa/web/lib/Transform/Filter.pm
A    elsa/web/lib/Transform/Count.pm
A    elsa/web/lib/Transform/Whois.pm
A    elsa/web/lib/Transform/Anonymize.pm
A    elsa/web/lib/Transform/Local.pm
A    elsa/web/lib/Transform/DNSDB.pm
A    elsa/web/lib/Transform/ScanMD5.pm
A    elsa/web/lib/Transform/Remote.pm
A    elsa/web/lib/Transform/Grep.pm
A    elsa/web/lib/Transform/CIF.pm
A    elsa/web/lib/Transform/GeoIP.pm
A    elsa/web/lib/Utils.pm
A    elsa/web/lib/Results.pm
A    elsa/web/lib/Web
A    elsa/web/lib/Web/GoogleDashboard.pm
A    elsa/web/lib/Web/Query.pm
A    elsa/web/lib/Web/Mobile.pm
A    elsa/web/lib/Web/Dashboard.pm
A    elsa/web/lib/Web/GoogleDatasource.pm
A    elsa/web/lib/AsyncMysql.pm
A    elsa/web/lib/Web.psgi
A    elsa/web/lib/Web.cgi
A    elsa/web/inc
A    elsa/web/inc/loading.gif
A    elsa/web/inc/startup.png
A    elsa/web/inc/utilities.js
A    elsa/web/inc/sprite.png
A    elsa/web/inc/open-flash-chart.swf
A    elsa/web/inc/main.js
A    elsa/web/inc/elsa.js
A    elsa/web/inc/swfobject.js
A    elsa/web/inc/custom.css
A    elsa/web/inc/dashboard.js
A    elsa/web/inc/admin.js
A    elsa/web/inc/wait.gif
A    elsa/web/inc/stats.js
A    elsa/web/inc/icon.png
A    elsa/web/inc/chart.js
A    elsa/web/inc/graphAnything.js
A    elsa/web/cron.pl
Exported revision 556.
get_elsa success
Executing update_web_mysql
Updating web MySQL, please ignore any errors for this section...
ERROR 1091 (42000) at line 1: Can't DROP 'action_params'; check that column/key 
exists
Finished updating MySQL
update_web_mysql success
!!!!!! IMPORTANT !!!!!!!!!
If you have a host-based firewall like IPTables running, remember to allow 
ports 80 (and/or 443) for the web server and ports 514 (syslog), 3306 (MySQL), 
and 3307 (Sphinx) for log nodes

Original comment by i...@pingas.org on 7 Dec 2012 at 7:14

GoogleCodeExporter commented 8 years ago

Looking at the httpd error log again, it looks like I'm missing *another* 
module... Is there a way to manually call the section of the install script 
that handles the perl modules?

[Fri Dec 07 14:16:07 2012] [error] [client 192.168.14.59] Error while loading 
/usr/local/elsa/web/lib/Web.psgi: Can't locate DateTime.pm in @INC (@INC 
contains: /usr/local/elsa/web/lib /usr/local/lib64/perl5 /usr/local/share/perl5 
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 
/usr/share/perl5 . /etc/httpd) at 
/usr/local/elsa/web/lib/Web/GoogleDatasource.pm line 11.\nBEGIN 
failed--compilation aborted at /usr/local/elsa/web/lib/Web/GoogleDatasource.pm 
line 11.\nCompilation failed in require at /usr/local/elsa/web/lib/Web.psgi 
line 14.\nBEGIN failed--compilation aborted at /usr/local/elsa/web/lib/Web.psgi 
line 14.\n
[Fri Dec 07 14:16:08 2012] [error] [client 192.168.14.59] Error while loading 
/usr/local/elsa/web/lib/Web.psgi: Can't locate DateTime.pm in @INC (@INC 
contains: /usr/local/elsa/web/lib /usr/local/lib64/perl5 /usr/local/share/perl5 
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 
/usr/share/perl5 . /etc/httpd) at 
/usr/local/elsa/web/lib/Web/GoogleDatasource.pm line 11.\nBEGIN 
failed--compilation aborted at /usr/local/elsa/web/lib/Web/GoogleDatasource.pm 
line 11.\nCompilation failed in require at /usr/local/elsa/web/lib/Web.psgi 
line 14.\nBEGIN failed--compilation aborted at /usr/local/elsa/web/lib/Web.psgi 
line 14.\n

Original comment by i...@pingas.org on 7 Dec 2012 at 7:17

GoogleCodeExporter commented 8 years ago

You ran build_web_perl already, which should have included DateTime, along with 
all other modules.  In the list of modules already upgraded, DateTime is not 
included, which leads me to believe you're using an old version of install.sh 
which isn't located in /usr/local/elsa/contrib/install.sh and is therefore not 
getting updated properly.  Can you try:

sh /usr/local/elsa/contrib/install.sh web update

Original comment by mchol...@gmail.com on 7 Dec 2012 at 7:37

GoogleCodeExporter commented 8 years ago

I was using the link to the script from the "Updating" section of the wiki.

ELSA is back up and has valid queries! Thank you for all your help!

Original comment by i...@pingas.org on 7 Dec 2012 at 7:47

GoogleCodeExporter commented 8 years ago

Ok, then this was a documentation bug that I'll fix immediately.  Thanks for 
letting me know where that error was, and I'm glad you're up and running.

Original comment by mchol...@gmail.com on 7 Dec 2012 at 8:04

Changed state: Done

IrcDirk / enterprise-log-search-and-archive

Can't see any queries... #85