cron.php - Errno 8: Undefined index: logstash

[profilernz]

I get this error if I run cron.php from the terminal:

/var/www/411/bin/cron.php [+] Scheduler: 1476998611 [+] Maintenance [+] Search Health except [NONE] Errno 8: "Undefined index: logstash" at [/var/www/411/phplib/Search/Elasticsearch.php:38] 0:[FOO\Elasticsearch_Search::getConfig() called at [/var/www/411/phplib/Search/Elasticsearch.php:66]] 1:[FOO\Elasticsearch_Search->isWorking() called at [/var/www/411/phplib/Scheduler.php:341]] 2:[FOO\Scheduler->health() called at [/var/www/411/phplib/Scheduler.php:146]] 3:[FOO\Scheduler->processSite() called at [/var/www/411/bin/cron.php:45]] except [NONE] Errno 8: "Undefined index: logstash" at [/var/www/411/phplib/ESClient.php:50] 0:[FOO\ESClient::getClient() called at [/var/www/411/phplib/Search/Elasticsearch.php:67]] 1:[FOO\Elasticsearch_Search->isWorking() called at [/var/www/411/phplib/Scheduler.php:341]] 2:[FOO\Scheduler->health() called at [/var/www/411/phplib/Scheduler.php:146]] 3:[FOO\Scheduler->processSite() called at [/var/www/411/bin/cron.php:45]] [+] Rollups [+] Searches [+] Reports [+] Summary [+] Autoclose [+] Cleanup

My config.php has no mention of a logstash index (since we don't use that default). The only ocurrence of "logstash" as an index seems to be on: phplib/Search/Logstash.php Also, not sure if related, but currently alerts don't fire unless someone presses execute in the webui. Any help much appreciated. Thanks!

[kiwiz]

Fixed! This should also resolve alerts not firing.

[profilernz]

Thanks! But maybe there's been a misunderstanding? We do use/need logstash searches, we just don't use the default "logstash-" index naming.

i.e. We use stats-, ossec- etc, but no logstash-: 'ossec' => [ 'hosts' => ['http://localhost:9200'], 'index_hosts' => [], 'sslcert' => null, 'index' => 'ossec-', 'date_based' => false, 'date_field' => '@timestamp', 'src_url' => null, ], 'stats' => [ 'hosts' => ['http://localhost:9200'], 'index_hosts' => [], 'sslcert' => null, 'index' => 'stats-', 'date_based' => false, 'date_field' => '@timestamp', 'src_url' => null, ],

It seems that with this commit, logstash searches have been disabled all together? I'm running v1.0.1 with your phplib/Search/Elasticsearch.php from the commit above. The error has disappeared after creating a new data.db, but now when we try to create a new search, we get "Null" instead of "Logstash" in the combo box. Thanks again!

[kiwiz]

Ah. The process for doing this is terrible atm: #3.

The alternative is to use the ECL search type. Ex: es:ossec somekey:somevalue.

[profilernz]

Thanks for the ECL workaround. But back to the issue, I don't want or need to add another dropdown option in the combo box. All I want is to add more indices (and none happen to be called logstash-). I had followed the instructions here: https://github.com/etsy/411/blob/master/docs/HowTo/NewESSource.md Which lead me to believe all I had to do is add another logstash block and call "things" something else. If there's no simple way to add a list of indices, maybe I should just go with 'index' => null, which should query them all? Thanks again!

[profilernz]

I'm now getting: /var/www/411/bin/worker.php [+] Worker [+] Worker: 1477430423 [+] Job count: 249 PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 near "LIMIT": syntax error' in /var/www/411/phplib/DB.php:78 Stack trace:

0 /var/www/411/phplib/DB.php(78): PDO->prepare('UPDATE jobs S...')

1 /var/www/411/phplib/Job.php(167): FOO\DB::query('UPDATE jobs S...', Array, 0)

2 /var/www/411/phplib/Worker.php(77): FOO\JobFinder::getAndLock('1', 1477430423)

3 /var/www/411/bin/worker.php(34): FOO\Worker->processSite(Object(FOO\Site), 1477430423)

4 {main}

thrown in /var/www/411/phplib/DB.php on line 78 err [411_Worker] Worker error site:[1] ret:[65280]

[profilernz]

Found https://github.com/etsy/411/issues/23

[kiwiz]

Oh, I see. You can't currently provide a list of indices, but 'index' => null would work. Feel free to open an issue for specifying multiple indices in a config block and we'll take a look when we have time.

[profilernz]

Issue https://github.com/etsy/411/issues/23 suggests replacing sqlite with mysql, but i wanted to stick with sqlite. For the record, I was able to fix the issue on Centos 7 by doing roughly the following:

yumdownloader --source sqlite or wget http://vault.centos.org/7.1.1503/updates/Source/SPackages/sqlite-3.7.17-6.el7_1.1.src.rpm (the source rpm for sqlite-3.7.17-8 was not available at the time for some reason, so i grabbed version "-6" above via wget)

yum install rpm-build redhat-rpm-config ncurses-devel readline-devel autoconf rpm -i sqlite-3.7.17-6.el7_1.1.src.rpm nano ~/rpmbuild/SPECS/sqlite.spec

at line 128 add -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 to the CFLAGS, i.e.:

export CFLAGS="$RPM_OPT_FLAGS -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -DSQLITE_DISABLE_DIRSYNC=1 -DSQLITE_ENABLE_FTS3=3 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_SECURE_DELETE=1 -DSQLITE_ENABLE_UNLOCK_NOTIFY=1 -Wall -fno-strict-aliasing"

rpmbuild -ba ~/rpmbuild/SPECS/sqlite.spec rpm -U --oldpackage ~/rpmbuild/RPMS/x86_64/sqlite-3.7.17-6.el7.centos.1.x86_64.rpm

[profilernz]

I can confirm the issue above also affects Fedora 24 (requiring a similar workaround), so Debian or Ubuntu might be an easier choice for those wanting to run 411 + sqlite.

[kiwiz]

I don't remember why I put that LIMIT 1 in the query, but it shouldn't be necessary as the UPDATE query is scoped to the PK... Fixed in e56a76f5a55d775c4ba3d3c9cb5e73af24ced10c!

[profilernz]

Nice! :)

[kiwiz]

Is this all good?

[kiwiz]

Closing. Feel free to reopen if this is not resolved!