ES 1.4 is EoL

[giovino]

Do we want look at supporting a newer version of ES given 1.4 hit EoL? (EoL is 18 months after release)

We'd first need to evaluate all the "breaking" changes listed in the change notes:

https://www.elastic.co/downloads/past-releases/elasticsearch-1-5-0 https://www.elastic.co/downloads/past-releases/elasticsearch-1-6-0 https://www.elastic.co/downloads/past-releases/elasticsearch-1-7-0 https://www.elastic.co/guide/en/elasticsearch/reference/2.0/release-notes-2.0.0.html https://www.elastic.co/guide/en/elasticsearch/reference/2.1/release-notes-2.1.0.html https://www.elastic.co/guide/en/elasticsearch/reference/2.2/release-notes-2.2.0.html https://www.elastic.co/guide/en/elasticsearch/reference/2.3/release-notes-2.3.0.html

If none can be identified, then test.

Anything else to do?

[giovino]

Using ES 2.3, get this perl error when trying to create a new token during install

 cd elasticsearch && make init )
make[1]: Entering directory `/home/giovino/massive-octo-spice/elasticsearch'
/usr/bin/curl -w "\n" -XPUT 'http://localhost:9200/_template/cif_observables/' -d @observables.json
{"acknowledged":true}
/usr/bin/curl -w "\n" -XPUT 'http://localhost:9200/_template/cif_tokens/' -d @tokens.json
{"acknowledged":true}
make[1]: Leaving directory `/home/giovino/massive-octo-spice/elasticsearch'
setting /etc/default/cif
setting up /etc/cif/cif-smrt.yml config...
Attribute (handle) does not pass the type constraint because: Validation failed for 'Search::Elasticsearch::Client::Direct' with value Search::Elasticsearch::Client::2_0::Direct=HASH(0x2abd4d0) at /usr/local/lib/perl/5.18.2/Mouse/Util.pm line 383.
    Mouse::Util::throw_error('Mouse::Meta::Attribute=HASH(0x1cf6a08)', 'Attribute (handle) does not pass the type constraint because:...', 'data', 'Search::Elasticsearch::Client::2_0::Direct=HASH(0x2abd4d0)', 'depth', -1) called at /opt/cif/bin/../lib/perl5/CIF/Storage/ElasticSearch.pm line 84
    CIF::Storage::ElasticSearch::_build_handle('CIF::Storage::ElasticSearch=HASH(0x21e02f0)') called at /opt/cif/bin/../lib/perl5/CIF/Storage/ElasticSearch.pm line 716
    CIF::Storage::ElasticSearch::token_new('CIF::Storage::ElasticSearch=HASH(0x21e02f0)', 'HASH(0x8b48c8)') called at /opt/cif/bin/cif-tokens line 212

[wesyoung]

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Storage/ElasticSearch.pm#L39 != Search::Elasticsearch::Client::2_0::Direct

which is what the new client is providing.

my advice, get to 1.7, then we can try moving to 2.x

[wesyoung]

i dunno that "isa => " really even needs to be defined either... probably take that out and it might just work.

[giovino]

this change gets through the installer but then when trying to make a query, you get this error:

[2016-05-13T10:44:15,301Z][29448][ERROR][CIF::Router:147]: [Request] ** [http://localhost:9200]-[400] [parse_exception] Failed to parse setting [timeout] with value [30000000] as a time value: unit is missing or unrecognized, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /opt/cif/bin/../lib/perl5/CIF/Storage/ElasticSearch.pm line 457. With vars: {'status_code' => 400,'body' => {'error' => {'type' => 'parse_exception','root_cause' => [{'type' => 'parse_exception','reason' => 'Failed to parse setting [timeout] with value [30000000] as a time value: unit is missing or unrecognized'}],'reason' => 'Failed to parse setting [timeout] with value [30000000] as a time value: unit is missing or unrecognized'},'status' => 400},'request' => {'qs' => {'timeout' => '30000000','size' => 50000},'ignore' => [],'method' => 'GET','body' => {'query' => {'filtered' => {'query' => {'match_all' => {}},'filter' => {'and' => [{'term' => {'observable' => ['example.com']}},{'or' => [{'term' => {'group' => ['everyone']}}]}]}}},'sort' => [{'reporttime' => {'order' => 'desc'}}]},'path' => '/cif.observables-%2A/_search','mime_type' => 'application/json','serialize' => 'std'}}

[wesyoung]

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Storage/ElasticSearch.pm#L442 ==> 300s

[giovino]

setting this to '300s' solves that problem but then you run into this..

[2016-05-13T10:54:11,058Z][29978][ERROR][CIF::Router:147]: [Request] ** [http://localhost:9200]-[400] [query_parsing_exception] [term] query does not support array of values, with: {"col":49,"line":1,"index":"cif.observables-2016.05.13"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /opt/cif/bin/../lib/perl5/CIF/Storage/ElasticSearch.pm line 457. With vars: {'body' => {'status' => 400,'error' => {'failed_shards' => [{'reason' => {'col' => 49,'line' => 1,'index' => 'cif.observables-2016.05.13','type' => 'query_parsing_exception','reason' => '[term] query does not support array of values'},'shard' => 0,'index' => 'cif.observables-2016.05.13','node' => '3_7s9r84SbuCVhyfm7ErOQ'}],'reason' => 'all shards failed','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'root_cause' => [{'reason' => '[term] query does not support array of values','index' => 'cif.observables-2016.05.13','type' => 'query_parsing_exception','line' => 1,'col' => 49}],'phase' => 'query','type' => 'search_phase_execution_exception'}},'status_code' => 400,'request' => {'mime_type' => 'application/json','serialize' => 'std','body' => {'query' => {'filtered' => {'filter' => {'and' => [{'term' => {'observable' => ['example.com']}},{'or' => [{'term' => {'group' => ['everyone']}}]}]},'query' => {'match_all' => {}}}},'sort' => [{'reporttime' => {'order' => 'desc'}}]},'qs' => {'timeout' => '300s','size' => 50000},'method' => 'GET','ignore' => [],'path' => '/cif.observables-%2A/_search'}}

which really means upgrading to 2.x is not trivial.

[wesyoung]

try 1.7...

[villain]

1.7 works fine

2.x has had a few major changes .. i would be surprised if there werent a few changes required to get it working

[wesyoung]

https://github.com/csirtgadgets/bearded-avenger/releases/tag/3.0.0a7

v3 should have better support for this; closing for now:

https://github.com/csirtgadgets/bearded-avenger/releases/tag/3.0.0a7