Pagination and Sorting of Results on the server

[nOkuda]

Searches with lots of results can take too long for end user devices to paginate and sort. Getting the server to do the work may improve the user experience.

[nOkuda]

(From Jeff)

Information the frontend will give to the API:

1. sort header
2. sort direction
3. number of rows per page
4. page number

Default settings would be sorted by score in descending order, 100 results per page, at page 0. It seems that the default should be to download all of the results, since that would be the functionality we want when the user decides to download the results.

[nOkuda]

Further research suggests that jumping by pages may not be optimal.

https://stackoverflow.com/a/5052898 recommends querying by the sort key of the last item appearing on the user's screen.

https://www.mongodb.com/blog/post/paging-with-the-bucket-pattern--part-1 recommends bucketing, which is to essentially pre-paginate the results, but this seems useful only if we limit the kinds of sorting permitted. That said, I can confirm that iterating through many little MongoDB documents takes much longer than I would like. I suppose iterating through a few large documents might alleviate this problem.

[nOkuda]

Another problem I've thought of: MongoDB might not sort by tags in quite the way we want them. We'll have to be extra careful to make sure they're sorted correctly.

[nOkuda]

Jumping by pages may not be too bad, since most users probably only look at the first page of results.

The tricky thing will be to order by something other than score. Tags and match features will need to be parsed in some way; will MongoDB be able to handle sorting in these ways?

[jeffkinnison]

Okay, so currently I only sort by score, source tag, and target tag. I couldn't figure out a sensible way to sort by match features that would also be transparent to the user.

With respect to the tags, I just sorted by string. It should hypothetically order things correctly, but you could also parse the number and sort by that directly.

[jeffkinnison]

I've also realized that the current response doesn't give me enough information to normalize scores or present the table correctly. I also need to know the max score computed by the search and the total number of results.