Support for Elasticsearch 5.x

[jorol]

Hi,

I've upgraded Elasticsearch to version 5.x. and catmandu import... and catmandu export... doesn't work anymore. The import fails without a warning.

Env:

• Ubuntu 16.04 64Bit
• Elasticsearch 5.2.1
• Catmandu 1.0306
• Catmandu::Store::Elasticsearch 0.0506

Error:

# import with catmandu
$ echo '{ "title":"My first blog entry","text": "Just trying this out...","date": "2014/01/01"}' | catmandu import -v JSON to ElasticSearch --index_name website --bag blog
imported 1 object
done

# count with catmandu
$ catmandu count ElasticSearch --index_name website --bag blog 
0

# export with catmandu
$ catmandu export ElasticSearch --index_name website --bag blog 
Oops! [Request] ** [http://localhost:9200]-[400] [illegal_argument_exception] No search type for [scan], called from sub Search::Elasticsearch::Client::5_0::Direct::scroll_helper at /home/jorol/perl5/perlbrew/perls/perl-5.22.2/lib/site_perl/5.22.2/Catmandu/Store/ElasticSearch/Bag.pm line 43. With vars: {'body' => {'error' => {'type' => 'illegal_argument_exception','root_cause' => [{'reason' => 'No search type for [scan]','type' => 'illegal_argument_exception'}],'reason' => 'No search type for [scan]'},'status' => 400},'status_code' => 400,'request' => {'serialize' => 'std','path' => '/website/blog/_search','qs' => {'size' => 100,'search_type' => 'scan','scroll' => '1m'},'ignore' => [],'mime_type' => 'application/json','method' => 'GET','body' => {'query' => {'match_all' => {}}}}}

# drop with catmandu
$ catmandu drop ElasticSearch --index_name website --bag blog 

# import via curl
$ curl -XPUT 'localhost:9200/website/blog/123?pretty' -H 'Content-Type: application/json' -d'
> {
>   "title": "My first blog entry",
>   "text":  "Just trying this out...",
>   "date":  "2014/01/01"
> }
> '
{
  "_index" : "website",
  "_type" : "blog",
  "_id" : "123",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}

# count with catmandu
$ catmandu count ElasticSearch --index_name website --bag blog 
1

# export with catmandu
$ catmandu export ElasticSearch --index_name website --bag blog 
Oops! [Request] ** [http://localhost:9200]-[400] [illegal_argument_exception] No search type for [scan], called from sub Search::Elasticsearch::Client::5_0::Direct::scroll_helper at /home/jorol/perl5/perlbrew/perls/perl-5.22.2/lib/site_perl/5.22.2/Catmandu/Store/ElasticSearch/Bag.pm line 43. With vars: {'status_code' => 400,'request' => {'qs' => {'search_type' => 'scan','size' => 100,'scroll' => '1m'},'ignore' => [],'path' => '/website/blog/_search','body' => {'query' => {'match_all' => {}}},'method' => 'GET','serialize' => 'std','mime_type' => 'application/json'},'body' => {'status' => 400,'error' => {'type' => 'illegal_argument_exception','reason' => 'No search type for [scan]','root_cause' => [{'type' => 'illegal_argument_exception','reason' => 'No search type for [scan]'}]}}}

[jorol]

It works when I add an _id and use the --key_prefix my_ option:

$ echo '{ "title":"My first blog entry","text": "Just trying this out...","date": "2014/01/01","my_id":"333"}' | catmandu -I ./lib import -v JSON to ElasticSearch --index_name website --bag blog --key_prefix my_
imported 1 object
done
$ catmandu count ElasticSearch --index_name website --bag blog 
1
$ catmandu export ElasticSearch --index_name website --bag blog --id 333
[{"text":"Just trying this out...","my_id":"333","date":"2014/01/01","title":"My first blog entry"}]

But I still get an error when I export the whole index:

$ catmandu export ElasticSearch --index_name website --bag blog 
Oops! [Request] ** [http://localhost:9200]-[400] [illegal_argument_exception] No search type for [scan], called from sub Search::Elasticsearch::Client::5_0::Direct::scroll_helper at /home/jorol/Devel/Dist/Catmandu-Store-Elasticsearch/lib/Catmandu/Store/ElasticSearch/Bag.pm line 43. With vars: {'status_code' => 400,'request' => {'qs' => {'size' => 100,'scroll' => '1m','search_type' => 'scan'},'body' => {'query' => {'match_all' => {}}},'path' => '/website/blog/_search','mime_type' => 'application/json','serialize' => 'std','ignore' => [],'method' => 'GET'},'body' => {'status' => 400,'error' => {'reason' => 'No search type for [scan]','type' => 'illegal_argument_exception','root_cause' => [{'reason' => 'No search type for [scan]','type' => 'illegal_argument_exception'}]}}}

[nics]

Key prefix is alsways needed with ES >= 2.0. I'll look into your last error.

[nics]

This is fixed in version 0.0507. Info about configuring the correct client for older versions of es has also been added to the pod