Email/Query Pagination 1 - handle pagination with negative position

[remk]

WHY

As a client I want to be able to get a long list of results in several operations. As a client i want to be able to start fetching the result only since a given negative index, to start fetching the result from the end of the list. It allows to get the last elements without knowing the total number of elements.

On this ticket we will only consider negative position.

    position: Int (default: 0) The zero-based index of the first id in the full list of results to return.

    If a negative value is given, it is an offset from the end of the list. Specifically, the negative value MUST be added to the total number of results given the filter, and if still negative, it’s clamped to 0. This is now the zero-based index of the first id to return.

    If the index is greater than or equal to the total number of objects in the results list, then the ids array in the response will be empty, but this is not an error.

request

    [[ "Email/query",{
      "accountId": "ue150411c",
      "position": -5
    }, "0" ]]

HOW

• accept a negative 'position' parameter in the query request.
• to get the index to start fetching add the position value to the total number of elements : for example for 100 elements and a given -5 position. Start fetching the 95th element.

DOD

• write an integration test demonstrating that when the client provide a negative position parameter, only the result since the total number of elements minus this position are returned in the response.

• write an integration test demonstrating that if the client provide a position parameter with an absolute value bigger than the total number of results then the response is empty and there is no error.

• run the tests against the distributed version of the contract

[rouazana]

write an integration test demonstrating that if the client provide a position parameter with an absolute value bigger than the total number of results then the response is empty and there is no error.

Not exactly. If the position is positive and superior to the number of elements, return 0 result. If the position is negative you have to:

Specifically, the negative value MUST be added to the total number of results given the filter, and if still negative, it’s clamped to 0. This is now the zero-based index of the first id to return.

So if the value is negative and the absolute value is superior to the number of elements, then you have to return from position 0.

[chibenwa]

This seems hard to be implemented with ElasticSearch....

What's the implementation strategy?

(Fetch all results then applying the negative offset seems like a non-effiocient way of doing this.)

[rouazana]

This seems hard to be implemented with ElasticSearch....

What's the implementation strategy?

(Fetch all results then applying the negative offset seems like a non-effiocient way of doing this.)

I think it gets close to the "total" discussion. If you have the total, a negative position is just total+position, and then you are back to a classic positive position.

[chibenwa]

total+position => you gonna likely destroy elastiSearch, having deep searches is an anti-pattern (as each shared needs to return top limit+position elements to be merged on the coordinator.)

BTW you would execute the same query two time: one time for getting the total, then the actual negative-turned-to-ppositive search. Sounds costly...

We had a call with @remk to discuss that very issue.

When a sort is specified, a cheaper alternative is to negate the sort then apply the limit (the result will be the same, but only "limit" items are going to be fetched. This turns the negative position as syntactic sugar for clients.

Rejecting negative position when no sort is specified could be wise, as it can bring the ES cluster to its knees, IMO.

[remk]

rejection of negative position handled here : https://github.com/linagora/james-project/pull/3796