scitran / core

RESTful API
https://scitran.github.io
MIT License
18 stars 18 forks source link

limit:-1 is failing on reasonable sized data sets #1003

Closed nagem closed 6 years ago

nagem commented 6 years ago

With a query that should return a reasonable size of results:

{
    "return_type":"file",
    "filters": [{
        "match":{"session._id":"58f177c11f4a0300130d5fe1"}}],
    "limit":-1
}

This error is returned:

TransportError: TransportError(500, u'search_phase_execution_exception', u'Result window is too large, from + size must be less than or equal to: [10000] but was [155984]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.')

It appears as if the initial count that sets the size for the actual query is returning a count of all files rather than a filtered count. Also knowing Elastic uses max_result_window, we should fail with a 400 - requested result set too large (or similar) when that original returned count is larger than the setting.