Elasticsearch query inconsistencies

[AGirin]

I think ES query is doing something wrong. I search kibana for messages:"invalid bid" and get some responses back.

Doing the same in 411 (same period of 6 hours give me nothing)

Not sure if this is something I am doing wrong.

[AGirin]

I see this error:

[error] [client xxxxxxx] except [NONE] Errno 8: "Undefined index: date_format" at [/var/www/411/phplib/Search/Elasticsearch.php:116] 0:[FOO\Elasticsearch_Search->constructQuery() called at [/var/www/411/phplib/Search.php:220]] 1:[FOO\Search->execute() called at [/var/www/411/phplib/Job/Search.php:56]] 2:[FOO\Search_Job->_run() called at [/var/www/411/phplib/REST/Searches.php:153]] 3:[FOO\Searches_REST->test() called at [/var/www/411/phplib/REST/Searches.php:57]] 4:[FOO\Searches_REST->POST() called at [/var/www/411/phplib/REST.php:108]] 5:[FOO\REST->route() called at [/var/www/411/htdocs/api/searches.php:6]], referer: http://411.test.test/searches/new?type=logstash

[kiwiz]

Add a date_format field in your config.php. Check out config_example.php for an example.

[AGirin]

Sorry, I think I am missing something. My config is set exactly how example shows:

I do not see any date_format field in the example...

[kiwiz]

You're completely right. The docs in the config were wrong. All fixed now.

I do appreciate that you're finding all these issues on master, though. Thanks! :]

[AGirin]

Well... I updated my configs and still nothing... What is interesting - I do get alerts from the rules but when I go to the rule and click Test button - I never get any results. And by the super fast response it feels like it does not search Elasticsearch at all. I selected time range as 2 days (and I saw errors in those two days from 411) and Test gave me "no results found" almost instantaneously (I wish my ES was that fast!) .

[AGirin]

Actually - now 411 is not working at all. No alerts, ES queries do not work. I think last update broke it somehow (I did use config_example to make sure I have new configuration in place).

[AGirin]

And I am getting this again:

[AGirin]

This is what I see now after the update:

/var/www/411/bin/cron.php [+] Scheduler: 1479236848 [+] Maintenance [+] Search Health PHP Fatal error: Call to undefined method ECL\Util::generateDateIndices() in /var/www/411/phplib/Search/Elasticsearch.php on line 76 err [411_Scheduler] Scheduler error site:[1] ret:[65280]

[AGirin]

I ran /usr/local/bin/composer install --no-dev --optimize-autoloader. Now I am getting this:

/var/www/411/bin/cron.php [+] Scheduler: 1479237171 [+] Maintenance [+] Search Health except [NONE] Errno 2: "array_key_exists(): The first argument should be either a string or an integer" at [/var/www/411/vendor/kiwiz/ecl/src/Util.php:36] 0:[array_key_exists() called at [/var/www/411/vendor/kiwiz/ecl/src/Util.php:36]] 1:[ECL\Util::exists() called at [/var/www/411/vendor/kiwiz/ecl/src/Util.php:19]] 2:[ECL\Util::get() called at [/var/www/411/vendor/kiwiz/ecl/src/Util.php:138]] 3:[ECL\Util::generateDateIndices() called at [/var/www/411/phplib/Search/Elasticsearch.php:76]] 4:[FOO\Elasticsearch_Search->isWorking() called at [/var/www/411/phplib/Scheduler.php:341]] 5:[FOO\Scheduler->health() called at [/var/www/411/phplib/Scheduler.php:146]] 6:[FOO\Scheduler->processSite() called at [/var/www/411/bin/cron.php:45]] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 32 bytes) in /var/www/411/vendor/kiwiz/ecl/src/Util.php on line 145 err [411_Scheduler] Scheduler error site:[1] ret:[65280]

[ghost]

I got the Same problem using master. I running within a docker container.

[kiwiz]

@AGirin Do you get results now?

[ghost]

@kiwiz cron is running on now without errors. Thanks!

[AGirin]

Well... No errors with cron anymore. No ES searches actually work. Test button has 0 results, Execute has 0 results. Something seriously broke after the last updates to config files.

At this point all my searches do not produce any alerts even though I see events in Kibana.

And yes - forgot about this one - after the last update I started getting alert (which I setup) - No Events in Elasticsearch. I know there are millions of events there so looks like 411 just cannot query ES at all.

[AGirin]

I am also getting this emails now after each time I restart 411 (was never happening before):

The logstash Search type is currently unavailable. Any Searchs of this type will not generate Alerts!
A followup email will be sent once this Search type is available.

And then

The logstash Search type is available!
Expect A LOT more Alerts in your future! 

[kiwiz]

@AGirin: I updated the index format for date_based indices. If you're using logstash indices, you should set 'index' => '[logstash-]Y.m.d'.

[AGirin]

I did that already. No luck.

[AGirin]

This is my config:

'logstash' => [ 'hosts' => ['http://testTest:9200'], 'index_hosts' => [], 'ssl_cert' => null, 'index' => '[logstash-]Y.m.d', 'date_based' => true, 'date_interval' => 'd', 'date_field' => '@timestamp', 'date_type' => null, 'src_url' => null, ],

[AGirin]

I set search to 7 days and the result comes back almost the moment I press the button test. No events. 411 stopped search ES. I am using 2.4 - was something changed recently? Like 5.0 support maybe which might have broken 411 for 2.4?

[AGirin]

I completely rebuilt the site from master - same result. Nothing works.

[kiwiz]

@AGirin How about now? I updated the composer dependencies (properly).

[AGirin]

Now everything is back online! Nice! Thanks a lot for looking into this!

[ghost]

Wohoo, Nice Work!