bug with pagination limitations per OpenSearch config

[tloubrieu-jpl]

🐛 Describe the bug

A request with start parameter = 10000 and limit=500 returns an error 500.

Appplication log says: 2022-07-19 22:21:45.796 ERROR 1 --- [p-nio-80-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is OpenSearchStatusException[OpenSearch exception [type=search_phase_execution_exception, reason=all shards failed]]; nested: OpenSearchException[OpenSearch exception [type=, reason=: Result window is too large, from + size must be less than or equal to: [10000] but was [10500]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Result window is too large, from + size must be less than or equal to: [10000] but was [10500]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]];] with root cause

📜 To Reproduce

Steps to reproduce the behavior:

1. Launch request:

curl --request GET 'https://pds.nasa.gov/api/search/1.0/products?q=pds:Identification_Area.pds:product_class eq "Product_Observational"&limit=500&fields=pds:Science_Facets.pds:discipline_name,lidvid,ops:Harvest_Info.ops:node_name&start=10000'

(url parameters need to be encoded)

2. Get error 500
3. See in registry-api logs:

2022-07-19 22:21:45.796 ERROR 1 --- [p-nio-80-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is OpenSearchStatusException[OpenSearch exception [type=search_phase_execution_exception, reason=all shards failed]]; nested: OpenSearchException[OpenSearch exception [type=, reason=: Result window is too large, from + size must be less than or equal to: [10000] but was [10500]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Result window is too large, from + size must be less than or equal to: [10000] but was [10500]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]];] with root cause

🕵️ Expected behavior

The size of the request is 500, we should not reach the limit given by opensearch.

📚 Version of Software Used

1.0.1

🩺 Test Data / Additional context

🏞Screenshots

🖥 System Info

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

---

🦄 Related requirements

⚙️ Engineering Details

[tloubrieu-jpl]

I am not adding that to the sprint-backlog yet because there are other bugs on Al's list.

[al-niessner]

@tloubrieu-jpl @jordanpadams

Yes, this is an error from opensearch that is more about opensearch limits/configurations than what registry can do:

OpenSearch exception 
   reason=: Result window is too large, from + size must be less than or equal to: [10000] but was [10500]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

I have noticed that opensearch has max limit size that bound between 500 and 1000. Our API allows for infinite or 2**31 -1 anyway which is like infinite.

At what threshold does registry make the choice of "large" window? opensearch rather arbitrarily chose 10000. Do we go smaller? Do we try to extract it from configuration? Do we set reasonable values in our API?

You want a restful API but opensearch scroll api requires caching and to hold that cache efficiently for start=10000 limit-500 and start=10500 limit=500 will require state.

[tloubrieu-jpl]

Hi @al-niessner , We could formlize the limitation in the API response but here the size of the window is 500 and the start page is 10000, at least as requested from the API. That used to work well (with elasticSearch and previous version of the API). I have difficulties to imagine that opensearch can not support that type of paginated requests.

This does not sounds related to the limit parameter which in the following example is 10:

curl --request GET 'https://pds.nasa.gov/api/search/1.0/products?q=pds:Identification_Area.pds:product_class eq "Product_Observational"&limit=10&fields=pds:Science_Facets.pds:discipline_name,lidvid,ops:Harvest_Info.ops:node_name&start=12000'

I am still getting a 500 error.

[al-niessner]

@tloubrieu-jpl

You should get an error with start=12000. As the error from opensearch that I cut out in this thread clearly states, start+limit < 10000. To get working, change the opensearch configuration parameter index.max_result_window..

Believe it or not, it has changed from 1e5 to 1e4 - explains why now and not before. Why it was changed not detailed but there is chatter, and annoyance, about it via google.

[tloubrieu-jpl]

@al-niessner Does that mean that we cannot get a record beyond 10000 in a result set ? I cannot believe that, this is a very standard pagination feature that I would like to use here. There must be something else.

[al-niessner]

@tloubrieu-jpl

Not until you shutdown opensearch, change index.max_result_window., then restart. Some folks suggest it can be changed and opensearcrh will catch the change without a restart. Not sure it will work without restart of opensearch.

This really has nothing to do with the registry. There are no checks with respect to size or limit beyond their min/max values by springswagger.

[tloubrieu-jpl]

ok, I am a bit disapointed, I found the same information on page https://opensearch.org/docs/1.1/opensearch/ux/ They say: Querying for pages deep in your results can have a significant performance impact, so OpenSearch limits this approach to 10,000 results.

I understand that it can be uncommon for human driven search to explore more than 10000 results but for querying datasets for science processing that might not be uncommon. We will need to find a solution for that.

[tloubrieu-jpl]

@jimmie could we change index.max_result_window in opensearch in production ? Or dev. There was no such limitation in elasticsearch but by default in opensearch the limit is 10000 which prevent us from being able to get more that 10000 results of a query even in multiple pages. The opensearch documentation says the performances will be bad if we go beyond 10000...

[jimmie]

@tloubrieu-jpl - I'd suggest changing it in dev first (en-delta and atm-delta) to see if there's a noticeable performance impact. Select your desired limit and file a ticket w/ DSIO: https://itsd-jira.jpl.nasa.gov/servicedesk/customer/portal/16

[tloubrieu-jpl]

That sounds good @jimmie , we'll need to populate the dev registry instance with something like the orex bundle (https://pds.nasa.gov/data/pds4/test-data/registry/orex.ovirs/) to have enough volume for testing this ticket and the performances

[al-niessner]

@jordanpadams @jimmie

Why do we think this is the right place? I just did a 'find . -type f -exec grep opensearch {} \; -print' and absolutely nothing appeared.

Whatever task/repo/SA/?? is resposible for starting opensearch needs to make this change. It does not appear to be this repo.

[jordanpadams]

@al-niessner I am fairly certain most things you can configure in opensearch and be configured via a config file we post when we setup the registry.

[jordanpadams]

@al-niessner maybe here?

https://github.com/NASA-PDS/registry-mgr/blob/main/src/main/resources/elastic/registry.json