support paging in cache.

[Sewdn]

Store the range of data that is already loaded for a query and merge subsequent data payloads in history.

This supports the use case for paged search-results (infinite scroll).

I'll deploy an example app shortly. I will also update readme and add tests if you're interested in this PR.

Kr, Pieter

[Sewdn]

I set up an example app to showcase paginated searching while also demoing how to use masonry layout manager properly in meteor and how to implement an infinite scroll: http://infinitesearch.meteor.com/ https://github.com/Sewdn/meteor-infinite-search

[arunoda]

Wow. That's awesome. I'd love to get this PR. Why don't you add a section in the README with a separate section for pagination. Make sure to link this demo.

[arunoda]

I have a simple way to handle this optional data and caching them. Rather than, hardly depend on skip and limit params. Why don't we stringify the options and cache data from that key. So, we cache for the searchText + JSON.stringify(options).

What do you think?

[Sewdn]

That's what I did first, have separate cache keys for every different request [query, skip, limit]. But the problem is the same documents will be cached multiple times if your paging is not consequent.

I mean, when you do subsequent requests (with the same query) like this [0,20], [5,10], [10,15], [15,20] only the first request should miss cache.

I still have to add logic to make sure the [skip, limit] options are modified with respect to what is already in cache. When you do subsequent requests like this [0,10] [5,15] [10,25], this should be modified to [0,10] [10,15] [15,25]

[arunoda]

May be we can do something like this.

We can use a global store for caching actual docs with the number of times they get cached. Then for "searchText" + stringifiedOptions only keep the id of the numbers only.

Will that going to save the issue?

On Sun, Apr 5, 2015 at 12:50 PM Pieter Soudan notifications@github.com wrote:

That's what I did first, have separate cache keys for every different request [query, skip, limit]. But the problem is the same documents will be cached multiple times if your paging is not consequent.

I mean, when you do subsequent requests (with the same query) like this [0,20], [5,10], [10,15], [15,20] only the first request should miss cache.

I still have to add logic to make sure the [skip, limit] options are modified with respect to what is already in cache. When you do subsequent requests like this [0,10] [5,15] [10,25], this should be modified to [0,10] [10,15] [15,25]

— Reply to this email directly or view it on GitHub https://github.com/meteorhacks/search-source/pull/14#issuecomment-89729964 .

[Sewdn]

Yes, then we only store the docs once, but we still make too much requests on the backend... Or do you think that's the responsibility of the devs using this package, to make sure they send the right queries to the SearchSource to avoid duplicate results?

[Sewdn]

Coming to think of it, I still have to add the [sort] to the cache key anyway, because that affects the entire set.

[arunoda]

Hmm. The thing, I don't want to take care of special handling for sort and limit fields. It works for mongodb, may be ES. But this needs to be work with any search engine.

On Sun, Apr 5, 2015 at 12:58 PM Pieter Soudan notifications@github.com wrote:

Coming to think of it, I still have to add the [sort] to the cache key anyway, because that affects the entire set.

— Reply to this email directly or view it on GitHub https://github.com/meteorhacks/search-source/pull/14#issuecomment-89730197 .

[Sewdn]

I think, when you want to support paging, you'll have to do some handling for sort, limit and skip options. This doesn't have to affect your backend. It's up to the developer using SearchSource to provide options for skip, limit and sort or not. They are entirely optional.

[roberto-belardo]

Wow this is awesome!

[drewproud]

This is great - I can't wait to use this.

[derwaldgeist]

+1

Is this extension already available somewhere? I was trying to set-up search-source with paging, but could not find a "skip" parameter.

[ShonM]

I spent some time with the fork from @Sewdn and rebased it against master as https://github.com/ShonM/search-source. I don't plan to maintain it, just wanted it for a quick side project and thought I'd link it here.