allow_partial_search_results setting must be passed as query-string parameter

[danielsnider]

The allow_partial_search_results setting must be passed as query-string parameter but this isn't possible with the tool that I'm using (ReactiveSearch). A global config option would be better for me or an option in the request body.

[matriv]

Hello @danielsnider,

The default value for this param can also be said globally through a cluster setting: search.default_allow_partial_results. Please check the documentation here.

[elasticmachine]

Pinging @elastic/es-search

[danielsnider]

@matriv Thanks so much! Do you know why it's not working?

Setting is set:

$ curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
    "persistent" : {
        "search.default_allow_partial_results" : "false"
    }
}
'
$ curl -X GET "localhost:9200/_cluster/settings"
{"persistent":{"search":{"default_allow_partial_results":"false"}},"transient":{}}

Then I GET string query with intentional syntax error "heart(":

{"query_string":{"fields":["*"],"query":"heart("}}

But I receive response 200 with even though the body is a clear error 😢:

{
  "responses": [
    {
      "error": {
        "root_cause": [
          {
            "type": "query_shard_exception",
            "reason": "Failed to parse query [heart(]",
            "index_uuid": "QrWhvbq2RIWijCAoXfFKhw",
            "index": "image"
          }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
          {
            "shard": 0,
...

How can we make the 200 response be the 400 or 500 error that is should be? Many thanks! This is a major blocker for me.

[jimczi]

@danielsnider this looks like a bug but I cannot reproduce, can you post the entire response and the version that you're using ?

[danielsnider]

@jimczi Thanks for taking a look.

I've tried ElasticSearch 6.7.1 and 6.6.0. This is how I start elastic:

sudo docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e ES_JAVA_OPTS="-Xms16g -Xmx16g" -v `pwd`/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /home/ubuntu/esdata:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:6.6.0

Full search request:

Request URL: http://192.168.136.128:9200/image/_msearch?
Request Method: POST
Status Code: 200 OK
Remote Address: 192.168.136.128:9200
Referrer Policy: no-referrer-when-downgrade
accept: application/json
content-type: application/x-ndjson
Origin: http://192.168.136.128:3000
Referer: http://192.168.136.128:3000/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36
x-requested-with: 771100

{"preference":"modality-list"}
{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"fields":["*"],"query":"heart("}}]}}]}},"highlight":{"pre_tags":["<mark>"],"post_tags":["</mark>"],"fields":{}},"size":0,"_source":{"includes":["*"],"excludes":[]},"from":0,"aggs":{"Modality.raw":{"terms":{"field":"Modality.raw","size":20,"order":{"_count":"desc"}}}}}
{"preference":"bodypart-list"}
{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"fields":["*"],"query":"heart("}}]}}]}},"highlight":{"pre_tags":["<mark>"],"post_tags":["</mark>"],"fields":{}},"size":0,"_source":{"includes":["*"],"excludes":[]},"from":0,"aggs":{"BodyPartExamined.raw":{"terms":{"field":"BodyPartExamined.raw","size":100,"order":{"_count":"desc"}}}}}
{"preference":"gender-list"}
{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"fields":["*"],"query":"heart("}}]}}]}},"highlight":{"pre_tags":["<mark>"],"post_tags":["</mark>"],"fields":{}},"size":0,"_source":{"includes":["*"],"excludes":[]},"from":0,"aggs":{"PatientSex.raw":{"terms":{"field":"PatientSex.raw","size":20,"order":{"_count":"desc"}}}}}
{"preference":"tagCloud"}
{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"fields":["*"],"query":"heart("}}]}}]}},"highlight":{"pre_tags":["<mark>"],"post_tags":["</mark>"],"fields":{}},"size":0,"_source":{"includes":["*"],"excludes":[]},"from":0,"aggs":{"descriptions.raw":{"terms":{"field":"descriptions.raw","size":200,"order":{"_term":"asc"}}}}}
{"preference":"results"}
{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"fields":["*"],"query":"heart("}}]}}]}},"highlight":{"pre_tags":["<mark>"],"post_tags":["</mark>"],"fields":{}},"size":12,"_source":{"includes":["*"],"excludes":[]},"from":0}

Full 200 response:

access-control-allow-credentials: true
access-control-allow-origin: *
content-encoding: gzip
content-length: 538
content-type: application/json; charset=UTF-8
Warning: 299 Elasticsearch-6.7.1-2f32220 "Deprecated aggregation order key [_term] used, replaced by [_key]"

{"responses":[{"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"image","node":"X6uClAg5SimP2fxx3L872g","reason":{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image","caused_by":{"type":"parse_exception","reason":"Cannot parse 'heart(': Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    ","caused_by":{"type":"parse_exception","reason":"Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    "}}}}]},"status":400},{"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"image","node":"X6uClAg5SimP2fxx3L872g","reason":{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image","caused_by":{"type":"parse_exception","reason":"Cannot parse 'heart(': Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    ","caused_by":{"type":"parse_exception","reason":"Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    "}}}}]},"status":400},{"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"image","node":"X6uClAg5SimP2fxx3L872g","reason":{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image","caused_by":{"type":"parse_exception","reason":"Cannot parse 'heart(': Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    ","caused_by":{"type":"parse_exception","reason":"Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    "}}}}]},"status":400},{"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"image","node":"X6uClAg5SimP2fxx3L872g","reason":{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image","caused_by":{"type":"parse_exception","reason":"Cannot parse 'heart(': Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    ","caused_by":{"type":"parse_exception","reason":"Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    "}}}}]},"status":400},{"error":{"root_cause":[{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"image","node":"X6uClAg5SimP2fxx3L872g","reason":{"type":"query_shard_exception","reason":"Failed to parse query [heart(]","index_uuid":"FqZfR415QIuajKOY32gXYg","index":"image","caused_by":{"type":"parse_exception","reason":"Cannot parse 'heart(': Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    ","caused_by":{"type":"parse_exception","reason":"Encountered \"<EOF>\" at line 1, column 6.\nWas expecting one of:\n    <NOT> ...\n    \"+\" ...\n    \"-\" ...\n    <BAREOPER> ...\n    \"(\" ...\n    \"*\" ...\n    <QUOTED> ...\n    <TERM> ...\n    <PREFIXTERM> ...\n    <WILDTERM> ...\n    <REGEXPTERM> ...\n    \"[\" ...\n    \"{\" ...\n    <NUMBER> ...\n    <TERM> ...\n    "}}}}]},"status":400}]}

Any ideas?

[danielsnider]

Same problem on both my laptop and production server. I can see in the elasticsearch log that the setting is getting set, but it's still not working.

[2019-04-16T17:14:02,049][INFO ][o.e.c.s.ClusterSettings  ] [1D5Dwct] updating [search.default_allow_partial_results] from [true] to [false]

[jimczi]

Ok I understand now, this is because you look at the status of the _msearch response. As you can see in the response array, the statuses for the internal requests are all set to 400. We discussed here if we should convey the error status from the internal responses to the main _msearch response but agreed that we should keep the current behavior of returning an error only if the parsing of the _msearch fails and not if the actual parsing of the internal queries failed in shards. We have the mechanism for _bulk queries where it is required to check the status of the individual requests to find the errors.

This is a major blocker for me.

Can you explain why ? The errors are correctly reported in the response. Also note that setting search.default_allow_partial_results has nothing to do with this behavior. If query parsing fails on all shards, the response for the individual request will always be a 400.

[danielsnider]

Crazy, so there's no way option to get an error HTTP status even when the response body is full of 400? This is a major blocker for me because these errors then do not bubble up to users, they see a loading screen forever. The tool I build on ReactiveSearch doesn't know what to do when it gets a 200 HTTP response who's body is all errors and no data. Could there be another setting? Standards say that 200 means "the response will contain an entity corresponding to the requested resource" and ElasticSearch is not behaving that way.

[jimczi]

Well the arguments why we return a 200 are all in the issue I linked. Your example is simple because you have the same query repeated multiple times but what should be the behavior if you have different types of errors in the response ? The reasoning for returning a 200 is that the problem is not on the _msearch but on the individual requests, you'd get an error if the _msearch query is malformed or if the dispatching of the queries failed in some way. I hope you don't mind if I close this issue as a duplicate of https://github.com/elastic/elasticsearch/issues/29169 and don't hesitate to add your thoughts there if you have counter arguments to the ones already exposed.

[danielsnider]

Thanks for helping me!! ❤

I agree with your explanation. I still think a new option would be very useful to enable an HTTP error status if any query had an error in a _msearch. ElasticSearch is breaking the HTTP standard! 🤮