Implement object-list methods

[ericelliott]

object-list

Treat arrays of objects like a db you can query. A single object from an object-list is called a record.

A common API for object collections

You may be scratching your head right now and wondering how this is different from Underscore, Lodash, or Rx Observables. Astute observations. The implementation will likely lean heavily on both Lodash and Rx Observables.

The difference is that this is intended to provide a universal facade for many types of object collections. It is an interface contract that will hopefully support a number of modular collection adapters. Read on for more details.

Current Status

Developer preview.

API Design Goals

Only a few of these design goals have been met in the current implementation, so read this section like everything is prefixed with "eventually..." See future.

• A common API for object collections (e.g. arrays of objects). Must have adapters: Array, Immutable.js:List, Rx Observable, Siren-Resource API via HTTP Fetch, Mongo client, Redis client.
• object-lists are immutable. Instead of mutating the source data, a new object-list will be returned.
• Completely modular. Enable require() at the function level similar to require('lodash/object/assign'), etc... Compatibility transforms for things that aren't required for hard dependencies should also be separate modules. Minimize browserify bundle weight.
• Node or Browser.
• Provide ES6 Array.prototype compatible API. Almost anything that takes an array as input should be able to take an object-list, as well, provided the API does not rely on array mutation.
• Use ES6 and make compiled ES5 version available on npm.
• Compatible with infinite streams and generators.
• Should be capable of sync or async. Add .subscribe() method for async results.

See Future.

What needs doing?

• [ ] Create checklist from method names (only partially complete)
• [ ] Create corresponding issues for each checklist item
• [ ] Comment on each issue with this meta-issues URL so that issue status is displayed below this ticket

  High Priority

• [ ] .push(record), .add(record) - Append the record to the list.
• [ ] .remove(obj) – Removes an item from the collection if it looks like the passed-in object (deepEqual).
• [ ] .removeWhere({key1: value, key2: value2...}) - Remove all objects that match the key:value where clauses.
• [ ] .distinct(predicate) - Returns only distinct records which satisfy the predicate function.
• [ ] .distinctWhere({key: value, key2: value2}) - Returns only distinct records which match all supplied where clauses.
• [ ] .sortBy({foo: 'descending'}, [fn]) – Sorts the list based on the property passed in using optional [].sort() compatible custom sort function.
• [ ] ES6 Array.prototype methods

  Medium Priority

• [ ] .at(index), .recordAt(index) – Returns the record at index.
• [ ] .removeSlice(startIndex, endIndex) – Filter out a range of indexes. Basically the opposite of .slice() (return the filtered list instead of the sliced out subset).

[benlesh]

This is interesting. How will this handle the asynchronous nature of Observables?

Assuming something like this: (I don't actually know the API)

var myObservable = Rx.Observable.interval(1000);
var index = 9;
var objList = objectList.from(myObservable);
var result = objList.at(index);

Is result a Promise? an Observable? It can't be a value, because index 9 might not have happened yet...

FWIW: I'm trying to get an effort going to provide a common interop API for reactive streams libraries like RxJS, Bacon and Most.js. It should look a lot like the Observable signature from @jhusain's ES7 async generator proposal. Which is the dual of the ES6 Iterable.

[ericelliott]

From future doc:

let myList = list({ list: apiResource, async: true })
  .reverse()
  .subscribe(onNext, onError, onCompleted);

[ericelliott]

Can you link to jhusain's proposal? Also from future doc:

• .toGenerator() - Return ES6 generator function if runtime supports it.

[mattpodwysocki]

@blesh since there are no blocking operators allowed at that point, it'd be best if it is still a wrapped value, eg Observable. Any aggregate operations should be "async" or wrapped values:

interface Observable<T> {
    sum() : Observable<T>
    sum<T, U>(selector: (item: T) => U) : Observable<U>
}

One thing that bugged me about the Streams API in Dart, in particular the Stream object is that any one and done operations are Future<T> or Promise values. That breaks the whole composition model and now you need to differentiate between Stream<T> and Future<T>. I'm sure @jhusain would have plenty to say about that particular issue.

But that breaks between Array and Observable because Array reductions provide a single unwrapped value.

[ericelliott]

:+1:

Yeah, we're going to be (mostly) ES6 array compatible so you can easily use an object-list almost anywhere you'd use an array, which is why you have to specify an option if you want async.

[benlesh]

Here's the link: https://github.com/jhusain/asyncgenerator#introducing-observable

[ericelliott]

Thanks! =)

[mattpodwysocki]

One thing that we should have is composition through some sort of flatMap or concatMap feature. This would enable such scenarios as several level deep queries. For Array values, it should be pretty easy of a Cartesian product.

for (let x of xs) {
  for (let y of f(x))
    yield y;
}

This enables scenarios such as this:

let videos = user.videoLists
  .flatMap(videoList =>
    videoList.videos.filter(video => video.rating === 5))

videos.forEach(onNext, onError, onCompleted)    

For Observable sequences, the water gets muddied in that you could base it upon a merge, meaning when a value from any stream is ready, it is sent down, which is preferable, or a concat which waits for the previous stream to end before emitting values.

I know there was some interest from @BrendanEich about getting flatMap or something akin to it into ES-Harmony.

[benlesh]

@mattpodwysocki I agree, and I think flatMap is important for interop.

[mattpodwysocki]

@ericelliott should we categorize the kinds of collections we are dealing with? For example, Set is different from Map is different from Array and of course Observable. Then perhaps you should also focus on say, immutable collections. Each will have a different contract.

[ericelliott]

Just implement equivalents of .concatAll() and .merge()?

I don't want to categorize with different contracts. Might as well just use RxJS, Lodash, etc... in that case.

The point of this is to present a unified API for all collection management needs. In cases where there's a distinction dependent on async, we'll figure out what makes the most sense on a case-by-case basis. I have a feeling those will be the exceptions rather than the rule.

[BrendanEich]

Everyone wants flatMap -- here's a sweet.js issue wanting it:

https://github.com/mozilla/sweet.js/pull/441#issuecomment-68223347

I think Matt's point about different contracts is good. Set and Map do not have the same contract, as sets have values but no keys, offer has/add/delete methods not get/set/delete, provide different iterators, etc.

[ericelliott]

:+1:

Should we avoid calling it flatMap to avoid potentially incompatible collision with the proposal that will almost certainly land in the spec, or should we just do it and break backward compat with ES-native when the dust settles?

[ericelliott]

I think Matt's point about different contracts is good. Set and Map do not have the same contract, as sets have values but no keys, offer has/add/delete methods not get/set/delete, provide different iterators, etc.

Yeah, I don't think values with no keys belongs in this API. We're specifically concentrating on collections of objects with key/value pairs because we need a good general-purpose utility belt for everything that makes heavy use of them (db, JSON APIs, event streams, etc...).

Those would probably be great separate modules sharing a lot of the same ideas, though! =)

[mattpodwysocki]

@ericelliott my comments were inline with having things like add and push being added but for most collections, that's simply not needed. Or you have differing APIs based upon whether immutable or mutable.

[JScheerleader]

This is cool.

[ericelliott]

@mattpodwysocki These are never mutable. Any change will return a new object-list.

Immutable.js has an interesting api for interim mutations that might be useful to emulate if we decide to allow mutations in some steps in the chain.

[ericelliott]

@JScheerleader Thanks! I'm already using and enjoying the current proof-of-concept. It should get really interesting when we start to implement the adapters. =)