magda-io / magda

A federated, open-source data catalog for all your big data and small data
https://magda.io
Apache License 2.0
493 stars 93 forks source link

Use `Search After` to access paginated result search over `max_result_window` limit #3524

Open t83714 opened 1 month ago

t83714 commented 1 month ago

Use Search After to access paginated result search over max_result_window limit

To page through a larger set of results, you can use the search API's from and size parameters. The from parameter defines the number of hits to skip, defaulting to 0. The size parameter is the maximum number of hits to return. Together, these two parameters define a page of results.

By default, you cannot use from and size to page through more than 10,000 hits. This limit is a safeguard set by the index.max_result_window index setting. If you need to page through more than 10,000 hits, use the search_after parameter instead.

https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after