search

abartov / bybeconv

Project Ben-Yehuda's content management system.
https://benyehuda.org/
Other
10 stars 5 forks source link

Solve the result window size limitation in ElasticSearch. #281

Closed abartov closed 3 months ago

abartov commented 3 months ago

Currently, only 10000 results are shown in all-works view (/works). It used to be possible to increase the result window size, but it stopped working for some reason, and anyway, a huge (~50000) result window is deprecated.

Figure out the recommended way to allow browsing/paging through the entire collection (currently approaching 50000 texts), and implement it.

damisul commented 3 months ago

I've added a quick fix, see attached PR.

As for the long-term solution. We can consider switching to search_after: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/paginate-search-results.html#search-after

But with this approach we cannot use current pagination system, when we use UI to switch to Nth page directly. Search_after implies different behavior, when we load first page with first 100 records and then there is a 'show next page' button which will load next 100 items when clicked. So in order to get to Nth page we'll need to do N-1 clicks.

Perhaps we can keep navigation by the first letter, but it will show only works starting from this letter (actually when clicking on letter it will do additional search with a letter and wildcard) and if there will be several pages we'll need to use that 'show next page' button too.

IMO this is quite significant change in functionality and UI. And API syntax needs to be updated too. @abartov , what do you think about such approach?