Some few additional features for displaying large sets of data

[ncortines]

Hi,

I wanted to propose the adding of the following features, which are aimed to support the displaying on large sets of data with additional sorting/filtering options:

• New indexName string parameter: Allows the user to fetch ALL records in the store using the specified index. Useful for getting lists sorted by different criteria
• New filter function parameter: Allows the user to filter out records which do not pass the filter function validation. Useful to extend the IndexedDB filtering capabilities, i.e. filtering with text wildcard or some complex condition
• Support for always and abort in the returned deferred/promise object

The example explains the use case:

var promise, // to reference the latest fetch promise
    loading = false, // a loading indicator
    movies = new Movies(); // which contains thousands of records

function loadMoviesCollection (textFilter, sortCriteria) {

    if (promise) {
        promise.abort(); // since movies collection contains thousands of records, the latest fetch operation may not be completed yet - abort it
    }

    movies.reset(); // clear the collection 

    loading = true; // loading again

    promise = movies.fetch({
        indexName: sortCriteria, // the sorting criteria may be different each time, i.e. sorting by 'name' or 'year', etc
        addIndividually: true, // we want a responsive UI
        filter: function (movie) {
            return movie.name.indexOf(textFilter) !== -1); // this kind of filtering is not supported in indexeddb and we don't want the overhead of adding these items to the collection. Filtering them in a collection with thousands of records would be too slow. This is faster and much more responsive
        }
    }).done(function () {
        console.log('done!'); // done successfully - all items loaded
    }).error(function () {
        console.log('error!'); // some error occurred while fetching
    }).always(function () {
        loading = false; // always
    });
}

loadMoviesCollection('The', 'name');

// Before promise is resolved user selects a different filtering/sort criteria

loadMoviesCollection('The', 'year');

Please let me know what you think of these enhancements.

Thanks!

[megawac]

The filter seems kind of questionable seeing the indexedDB probably came from a previous collection. I would probably use something like a VirtualCollection myself and load in all the data, but I don't know what you're doing so that may not make sense.

+1 anyway

[ncortines]

Please let me elaborate a bit more on the example given.

The Movies collection contains thousands of records, say 100.000. Bringing all the data to a (virtual) collection is memory to later re-sort it every time the user changes the sorting criteria, i.e, "name", "year", etc, is not efficient. It would take several seconds or minutes before being able to display the first record to the user.

It makes more sense to leverage IndexedDB's built-in sorting capabilities, defining one index per each of the sorting capabilities you want to expose to the users in order to display the data. In this way every time the user changes the sorting criteria reloading the list with sorted records is almost immediate.

The filter function speeds up the thing and saves memory, since it removes the overhead of adding unwanted models to the collection.

[megawac]

Alright, sounds reasonable, I've used the model to store up to 10,000 points for a map and didn't have issues but I can see it being a problem

[ncortines]

This is how it works: http://codepen.io/ncortines/pen/ZYGqby

[megawac]

Cool, I'm definitely for this, @ncortines could you write a couple test cases

[ncortines]

@megawac, sure, I will. Only one more thing to discuss is this indexName parameter, which tells the plugin to fetch all the store records using a specific index. I will also need to add an additional parameter to specify the direction, i.e. indexDirection (ascending/descending). I came up with these because I couldn't figure a better way to merge with the existing conditions syntax, which does not seem to allow users to force the plugin to use a specific index. Would you like to propose some alternatives to indexName and indexDirection options which play better with the current options syntax?

Thanks

[megawac]

Could also use order ala lodash, any of them are fine IMO

On Thu, Dec 4, 2014 at 10:28 AM, Juan Cortines notifications@github.com wrote:

@megawac https://github.com/megawac, sure, I will. Only one more thing to discuss is this indexName parameter, which tells the plugin to fetch all the store records using a specific index. I will also need to add an additional parameter to specify the direction, i.e. indexDirection (ascending/descending). I came up with these because I couldn't figure a better way to merge with the existing conditions syntax, which does not seem to allow users to force the plugin to use a specific index. Would you like to propose some alternatives to indexName and indexDirection options which play better with the current options syntax?

Thanks

— Reply to this email directly or view it on GitHub https://github.com/superfeedr/indexeddb-backbonejs-adapter/pull/73#issuecomment-65648393 .

[ncortines]

@megawac, please have a look. The list of new arguments looks like this:

• sort, which is an object with 2 properties:
  • index, mandatory, which is an string and stands for the original indexName parameter
  • order, optional, defaults to 1 (ascending). Setting it to -1 causes cursor to run on descending order
• filter, optional filtering function which must return either true (filter IN) or false (filter OUT). Can be combined with the new sort options as well as with any of the others (conditions, range...)

Additionally the returned promise has a new method:

• abort, causes the current fetch operation referenced to fail immediately. No more records are added to the collection.

I've added a few unit tests to cover these new features.

Thanks!

[megawac]

Ping @julien51 looks good, want to release a 0.1?

[julien51]

Would any of you be interested in becoming the maintainer? My time is very limited these days... and this is clearly putting a drag on this project.

[megawac]

As I mentioned earlier, I'd be happy to - that said would I be free to do some reasonably major refactors, I collab on several backbone data sync modules and it would be nice if the integration more or less mirrored Backbone.hoard

[julien51]

Yes, I'd let you do changes... even major because I trust you'll favor the library and its users over specific use cases. I'll add you right now.