Search results and pagination? Maybe new types of stores

[idolize]

I've been thinking about how to handle performing searches in Marty - specifically server-side searches.

Right now, this kind of searching doesn't play very well with the fetch() API and the basic concept of stores in Marty. (This is somewhat related to issue #111)

When searching with basic Marty stores:

• We can’t just check the store's cache because we don’t know if the cache has all relevant items
• However, we could optimistically show SOME local cached results while the remote call waits
  • This clearly doesn't work with the simplistic logic of fetch(), which really only consideres the case where locally() is completely empty (as opposed to partially containing the results we want)
  • This also doesn't work well with pagination, which is reason enough for me to not concern myself with it
• We could just avoid using the Store and fetch() altogether (getting rid of the cache layer) and simply return the results of the API call, but this isn't so great (for one thing it means we can't really use the awesome Container methods)

So can we cache this, or do we just ditch fetch() altogether?

New idea: multiple types of stores

• Content stores keep the actual cache of items fetched - similar to all the existing Marty store examples
  • Ex: state[id] = { id: 'id1', name: 'myObj' }
• Search Result stores keep just simple lists of IDs rather then whole objects
  • Ex: state[searchTerms][pageNo] = [ ‘id1’, ‘id2’, ‘id3’ ]
  • This way, we can update the content of any item whenever we query for new results, but we can also reuse caches of paginated results (when paging backwards, for example)
• Thus, whenever a new search is requested (or a new page of the search results is requested), the basic flow is as follows:
  1. First the Search Result store is checked to see if state[searchTerms][pageNo] exists (if so it is returned)
  2. If not, then the request to the server is made (Query in Marty terms)
  3. The server's response is parsed to pull out the IDs of the paginated items in a separate array and cache those in a Search Result store while any actual content items are either added or updated in place inside the corresponding Content store
• See this project for the inspiration

How feasible is this with the existing fetch() API? Is there an easy way to express the relationship between the Search Result store and the backing Content store (so I can easily fetch the list of users for a page after merging the IDs from the List store with the backing data from the Content store)?

[jhollingworth]

It might be useful to explian a little the theory behind the fetch API. There are 2 main reasons for it:

1. It's a way of describing how to get some specific state locally and remotely (if its not found locally).
2. Because a component might call a store multiple times (e.g. because another store change) we need a way to prevent us making additional remote queries when the previous one hasn't completed. Thus fetch creates a sort of lock (using the id as the key) which is re-opened when the remote promise completes (or locally returns a value).

These two things are not overly difficult to implement yourself and you could easily forgoe the fetch API altogether if you wanted to. Your proposed idea makes a lot of sense and you could get it to work happily with containers if you return a fetch result from the store:

var fetch = require('marty/fetch');

var pending = fetch.pending();
var failed = fetch.failed(new Error());
var done = fetch.done({ foo: 'bar' });

As an alternative for you to considering, you could have your locally always return a value but have have a side effect to get missing values. This is not an accurate example but should give you a rough idea of what I'm thinking:

class UserStore {
  getUsers(options) {
    var from = options.skip;
    var to = from + options.take;

    return this.fetch({
      id: `users-${from}-${to}`,
      locally() {
        var users = _.slice(this.state, from, to);

        // If there are any users missing from the list
        if(_.all(users, Boolean) && !someSortOfLockOverThisQuery) {
          // Go get the missing users
          UserQueries.getUsers(options);
        }

        return users;
      }
    })
  }
}

UserStore.getUsers({
  skip: 20,
  take: 10
});

[jhollingworth]

Hey, I'm going to close this for now. Please re-open if you feel my comment hasn't fully answered your question

[idolize]

@jhollingworth That's fine to close this; it was pretty vague anyway.

I do like your idea of returning some cached data in addition to kicking off a Query to fetch the latest data. Can you explain more about the someSortOfLockOverThisQuery value?