search

abartov / bybeconv

Project Ben-Yehuda's content management system.
https://benyehuda.org/
Other
10 stars 5 forks source link

Migrate to search_after in ElasticSearch, keeping A-Z functionality #283

Closed abartov closed 5 months ago

abartov commented 6 months ago

Damir wrote:

As for the long-term solution. We can consider switching to search_after: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/paginate-search-results.html#search-after

But with this approach we cannot use current pagination system, when we use UI to switch to Nth page directly. Search_after implies different behavior, when we load first page with first 100 records and then there is a 'show next page' button which will load next 100 items when clicked. So in order to get to Nth page we'll need to do N-1 clicks.

Perhaps we can keep navigation by the first letter, but it will show only works starting from this letter (actually when clicking on letter it will do additional search with a letter and wildcard) and if there will be several pages we'll need to use that 'show next page' button too.

IMO this is quite significant change in functionality and UI. And API syntax needs to be updated too. @abartov , what do you think about such approach?

Originally posted by @damisul in https://github.com/abartov/bybeconv/issues/281#issuecomment-2010493515

abartov commented 6 months ago

Thank you for researching this. I now recall reaching a similar conclusion reading about search_after, but thinking I must have misunderstood and planning to get back to it...

So, it seems clear we cannot extend the current system forever, and it is already increasing memory usage uncomfortably for our server size. Therefore, we will need to migrate to search_after at some point, but perhaps it doesn't have to happen at once.

When we do, though, I do want to keep the jump-to-letter functionality (even if we lost the less-meaningful jump-to-page-number one). To do this, we can keep a cached pre-calculated table of what to search_after for each letter for the default all-works search (the relatively common case where we would want to quickly browse to a particular letter), and for filtered searches (which we can't pre-calculate), to have an on-demand feature, i.e. a "jump to specific letter" button which, when pressed, would run queries and then display the letter bar for the particular result set.

What do you think?