Remove batching (was: Index scan does not work with > 1 page of results)

[aboodman]

At least that is my guess. Customer showed me a case of an index scan ilooping requesting pages from repc (e.g., calling the scan() rpc over and over). There were 2k items in the result set.

I bet what is happening is that the js wasn't updated to pass the correct start key in the case of secondary index scanning (vs primary index scanning).

[aboodman]

I looked at this briefly. What's happening is that the JS SDK passes the last 'key' it saw back to the scan rpc that it saw to get the next page of results: https://github.com/rocicorp/replicache-sdk-js/blob/master/src/scan-iterator.ts#L127

However, repc doesn't return keys in the case of scan: https://github.com/rocicorp/repc/blob/master/src/db/read.rs#L122

This was because I realized I can't decode that data structure I was using to get the key back out.

I have an idea how to fix this @phritz, we can talk about it in person or I'll write it down later.

[aboodman]

I had a complicated idea for how to fix this, but here's an easier one:

@arv what if we just remove batching from scan completely? The problem is we keep fetching the same batch from scan over and over. But we don't need batches anymore because of the way scan was changed to pass results back to JS one by one. We can just stop iterating at any point and implement limit just in the bindings.

WDYT?

[arv]

SGTM

[phritz]

Q that I think I saw mentioned on slack or somewhere: if we are going to remove batch are we positive that the way we pass individual results back to js is compatible with moving to sw?

[aboodman]

I think it's compatible in the sense that we can implement it, but I'm not sure if at the end of the day w/ sw it ends up making sense to leave it batched or remove it.