ch04/on_time_flask_template.py returns pyelasticsearch.exceptions.ElasticHttpError: (400, 'illegal_argument_exception')

[markucsb]

127.0.0.1 - - [07/Dec/2017 21:13:39] "GET /flights/search/?debugger=yes&cmd=resource&f=console.png HTTP/1.1" 200 - ImmutableMultiDict([('Carrier', ''), ('FlightDate', '2015-02-01'), ('TailNum', ''), ('Origin', 'JFK'), ('Dest', 'LAX'), ('FlightNum', '')]) QUERY 2015-02-01 JFK LAX {"from": 0, "sort": [{"FlightDate": {"order": "asc", "ignore_unmapped": true}}, {"DepTime": {"order": "asc", "ignore_unmapped": true}}, {"Carrier": {"order": "asc", "ignore_unmapped": true}}, {"FlightNum": {"order": "asc", "ignore_unmapped": true}}, "_score"], "size": 15, "query": {"bool": {"must": [{"match": {"FlightDate": "2015-02-01"}}, {"match": {"Origin": "JFK"}}, {"match": {"Dest": "LAX"}}]}}} GET /agile_data_science/_search [status:400 request:0.001s] 127.0.0.1 - - [07/Dec/2017 21:13:51] "GET /flights/search/?Carrier=&Origin=JFK&Dest=LAX&FlightDate=2015-02-01&TailNum=&FlightNum= HTTP/1.1" 500 - Traceback (most recent call last): File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 2000, in call return self.wsgi_app(environ, start_response) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1991, in wsgi_app response = self.make_response(self.handle_exception(e)) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1567, in handle_exception reraise(exc_type, exc_value, tb) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise raise value File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app response = self.full_dispatch_request() File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise raise value File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request rv = self.dispatch_request() File "/home/ubuntu/anaconda/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request return self.view_functionsrule.endpoint File "/home/ubuntu/Agile_Data_Code_2/ch04/web/on_time_flask_template.py", line 153, in search_flights results = elastic.search(query,index='agile_data_science') File "/home/ubuntu/anaconda/lib/python3.5/site-packages/pyelasticsearch/client.py", line 93, in decorate return func(*args, query_params=query_params, kwargs) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/pyelasticsearch/client.py", line 859, in search return self._search_or_count('_search', query, kwargs) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/pyelasticsearch/client.py", line 836, in _search_or_count query_params=query_params) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/pyelasticsearch/client.py", line 285, in send_request self._raise_exception(status, error_message) File "/home/ubuntu/anaconda/lib/python3.5/site-packages/pyelasticsearch/client.py", line 299, in _raise_exception raise error_class(status, error_message) pyelasticsearch.exceptions.ElasticHttpError: (400, 'illegal_argument_exception') 127.0.0.1 - - [07/Dec/2017 21:13:51] "GET /flights/search/?debugger=yes&cmd=resource&f=jquery.js HTTP/1.1" 200 - 127.0.0.1 - - [07/Dec/2017 21:13:51] "GET /flights/search/?debugger=yes&cmd=resource&f=style.css HTTP/1.1" 200 - 127.0.0.1 - - [07/Dec/2017 21:13:51] "GET /flights/search/?debugger=yes&cmd=resource&f=debugger.js HTTP/1.1" 200 - 127.0.0.1 - - [07/Dec/2017 21:13:51] "GET /flights/search/?debugger=yes&cmd=resource&f=console.png HTTP/1.1" 200 - 127.0.0.1 - - [07/Dec/2017 21:13:52] "GET /flights/search/?debugger=yes&cmd=resource&f=ubuntu.ttf HTTP/1.1" 200 - 127.0.0.1 - - [07/Dec/2017 21:13:52] "GET /flights/search/?debugger=yes&cmd=resource&f=console.png HTTP/1.1" 200 -

[rjurney]

@markucsb I am able to reproduce locally. I am looking into it, but it seems to be this issue: https://github.com/elastic/elasticsearch-py/issues/718

[ledgerW]

@markucsb @rjurney FWIW, I encountered this same issue (also in ch06 for /airplanes search), and was only able to circumvent the error by commenting out the sort key-value pair in the elasticsearch query DSL (like shown below).

Also, FWIW... a) I get the same errors when using the official elasticsearch client vs. pyelasticsearch, and b) while the 'ignore_unmapped' parameter gives an illegal argument error, switching to the 'unmapped_type' parameter gives a search_phase_execution_exception, which seems to indicate an issue with the search results (others report seeing this error when receiving too many results, not having enough memory, or a sorting "tie" or "collision" or some kind)

Build the base of our elasticsearch query query = { 'query': { 'bool': { 'must': []} },

'sort': [

#  {'FlightDate': {'order': 'asc', 'ignore_unmapped' : True} },
#  {'DepTime': {'order': 'asc', 'ignore_unmapped' : True} },
#  {'Carrier': {'order': 'asc', 'ignore_unmapped' : True} },
#  {'FlightNum': {'order': 'asc', 'ignore_unmapped' : True} },
#  '_score'
#],
'from': start,
'size': config.RECORDS_PER_PAGE

}

[rjurney]

@ledgerW @markucsb I have downgraded Elasticsearch for both the Vagrant and EC2 images. Can you try again with the new build setup? This should be resolved, but if not I'll try again. Thanks!

[10843]

@rjurney What version of ElasticSearch did you downgrade to?

Thx

[rjurney]

@10843 5.X but it doesn't fix things for me. Working on this.

[10843]

I was able to get pyelasticsearch to work with elasticsearch 5.0.0. I downgraded the library to use 5.0.0 versions as well. I did have to alter your elasticsearch query syntax to exclude the “ignore_unmapped” tags as they are not supported in that version.

I am now working on updating the mappings to allow retrieval of values in text columns.

Eric

From: Github Notifications notifications@github.com Reply-To: rjurney/Agile_Data_Code_2 reply@reply.github.com Date: Monday, April 30, 2018 at 8:14 PM To: rjurney/Agile_Data_Code_2 Agile_Data_Code_2@noreply.github.com Cc: Eric Theis eric_theis@gensler.com, Mention mention@noreply.github.com Subject: Re: [rjurney/Agile_Data_Code_2] ch04/on_time_flask_template.py returns pyelasticsearch.exceptions.ElasticHttpError: (400, 'illegal_argument_exception') (#58)

@10843https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2F10843&data=02%7C01%7Ceric_theis%40gensler.com%7C5b4793f9cfad4b30877708d5af1199c8%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636607412526284898&sdata=drJdLmixDNSNSncmNCJ4%2FWfAtw8Sw%2BG3KpWCLN1OHlk%3D&reserved=0 5.X but it doesn't fix things for me. Working on this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frjurney%2FAgile_Data_Code_2%2Fissues%2F58%23issuecomment-385589520&data=02%7C01%7Ceric_theis%40gensler.com%7C5b4793f9cfad4b30877708d5af1199c8%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636607412526284898&sdata=DUoBERubjf3BipwTtwyzuZ9KdFEjNhiiHAnZwnDJv1M%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABqqvE2CVMoP8ZDAAgmNDTuOCsIi0JCfks5tt9L9gaJpZM4Q6NHX&data=02%7C01%7Ceric_theis%40gensler.com%7C5b4793f9cfad4b30877708d5af1199c8%7C94a74758f2ff413c9f705725701b8d02%7C0%7C0%7C636607412526284898&sdata=2HptjB6CIU4y%2BrSu9%2FIMRQ7JaBMznSylwhgYuHvf0Ok%3D&reserved=0.

[rjurney]

Now using elasticsearch 5.6.1 to resolve this problem.