Collection API and pickCombine/pickMerge

[staltz]

I pushed to the collection branch an experimental feature where we borrow an API from Cycle Collection and improve performance under the hood.

The gist is:

Create many instances of a component using collection

function MyList(sources) {
  // ...

  // Expects sources.onion.state$ to be a Stream<Array<ItemState>>
  const instances$ = collection(Item, sources, itemState => itemState.key)
  // the above is the same as
  //const instances$ = collection(Item, sources)

pickCombine('DOM') is pick('DOM') + mix(xs.combine)

const childrenvnodes$ = instances$.compose(pickCombine('DOM'))

pickMerge('onion') is pick('onion') + mix(xs.merge)

const childrenreducer$ = instances$.compose(pickMerge('onion'))

---

Note: this is an experiment. That's why it's in a branch.

The motivation for this was to make onionify faster for large amounts of child components from an array. The main solution for this was pickCombine and the internal data structure of "instances" which collection builds. pickCombine is a fusion of pick+mix to avoid a combine, a flatten, and a map, and does all of these together and takes shortcuts to avoid recalculating things in vain. collection was sort a necessity from two perspectives: (1) easier than creating and isolating item components by hand, (2) with this performance improvement it would have become even harder to do it by hand.

Check the advanced example.

We're looking for feedback in order to find the most suitable API that feels quick to learn for the average programmer, while solving the problems below too. In other words, the challenge here is API design for developer experience. The challenge is not "how" to solve the technical problem.

Open problems/questions:

• [x] Solved thanks to a PR by @abaco. A child component may get double updates since it is onionified as well as the parent is. When the child component sends out a reducer to the onion sink, it will update its onion source state, but also the parent onion source state. We need to find a way to make the child avoid the second update since its redundant.
• [x] Update or replace the test "should work with an isolated list child with a default reducer"
• [x] How do we allow lenses on individual items, like we can do e.g. in example mixed?
• [x] collection() does a lot under the hood (creates the instances data structure, calls onionify, picks the key from each item state, creates item state lenses, onionifies each child component, etc). Do we want to have heavy configuration with many arguments, or do we want to break that down into other helper functions? And how?
• [x] How to avoid fragility to object equality issues? (read the thread)
• [x] How to keep the feature where we can pass a channel name instead of "onion"
• [x] This change expects item state to have a unique key. Do we still want to allow keyless items in a list?

Checklist before final release:

• [x] Rewrite official cycle examples
• [x] Rewrite todomvc
• [x] Rewrite matrixmultiplication
• [x] Write JSDocs for asCollection, pickCombine, pickMerge
• [x] asCollection to support state$ when it's a stream of objects
• [x] Implement CollectionSource
• [x] Rename asCollection to toCollection
• [x] Gather more feedback
• [x] Fix bug described by ntilwalli
• [x] Add tests for Cycle Unified
• [x] Check issue ntilwalli raised about isolateEach
• [x] Fix issue ntilwalli raised about RxJS support
• [x] Check issue ntilwalli raised about using toCollection without wrapper List component
• [x] Check issue atomrc raised about toCollection()
• [x] Check issue jvanbruegge raised about initial state
• [x] Update readme.md docs
• [x] Publish

[jvanbruegge]

1. Maybe by passing a factory from (childstate, index) => lens
2. I dont mind, i always have ids. Is the name customizable? Because passing in a lens just to comfort the internals doesnt feel right
3. passing a function that composed of the sub parts. Onionify exports the single steps for you to compose yourself (together with custom logic) and internally defaults to the standard composition
4. No idea (yet)

[staltz]

Thanks @jvanbruegge for the feedback. Yeah one thing I considered was breaking up collection into 2 or 3 functions, and specially so that array$ is a function input. Currently it's non-obvious that collection(Comp, sources) uses sources.onion.state$ as the array$.

[staltz]

Do we want to have heavy configuration with many arguments, or do we want to break that down into other helper functions? And how?

I tried to do this just now and didn't get progress. I tried splitting into diff(array$, getKeyFn) and collection(diff$, Comp, sources) but it turns out this second part also needed the array$, so that didn't work.

I think I'll just stick with collection(Comp, sources) which is the same that Cycle Collections uses. It was at least proved to work well with people who've used that lib.

Now I'm trying to see how to allow configurability such as

How do we allow lenses on individual items, like we can do e.g. in example mixed?

[staltz]

So far the candidate API is

function collection<Si>(Comp: (so: any) => Si,
                        sources: any,
                        lens: (key: string) => Lens<Array<any>, any> = defaultInstanceLens,
                        getKey: any = defaultGetKey): Stream<Instances<Si>>

It's a bit ugly as a 4-arg function, and I tried to join the 3rd and 4th as one thing but it doesn't seem possible since the lens factory needs a key as argument, and to generate that key we need getKey.

This API would work for all the use cases we listed, but I don't know, doesn't feel so elegant in my opinion.

[staltz]

I think I have good news. We don't need the 4-arg API, we can have just collection(Comp, sources, getKey) because we can solve...

How do we allow lenses on individual items, like we can do e.g. in example mixed?

through the list lens, not the list item lens. I wrote mixed example with collection(): https://github.com/staltz/cycle-onionify/commit/ac670c3889e62817042205eb43cde21628788a8f

So the only question left to answer is:

Do we still want to allow keyless items in a list?

And a new question I would add for future-proofing:

Are collections always going to be arrays? Can we leverage ES6 Set, Map? Should we?

[jvanbruegge]

how to specify the array$? Most of the time the array is a property on a state object.

I dont need keyless ones, but i want to be able to rename key to id or anything else.

Not sure about set or map

[staltz]

Are collections always going to be arrays? Can we leverage ES6 Set, Map? Should we?

Answering myself, I think the use case for a collection is driven by how it should be displayed. Most collections of UI widgets are lists. In some cases you have 2D grids, but those can also be represented as arrays. I find it quite hard to imagine another case, but if needed, it seems possible to build mapCollection (uses ES6 Map) as a separate library, if needed.

Also I just made a commit where we use some Set and Map for the internal data structures. The API is still exactly the same.

[staltz]

how to specify the array$? Most of the time the array is a property on a state object.

Yeah, collection assumes sources.onion.state$ is an array$. This is something that just needs to be learned. I tried different alternative APIs, where the wiring with array$ is explicit, but it becomes weird and ugly, because in the end we also need the whole sources object.

dont need keyless ones

👍

but i want to be able to rename key to id or anything else.

You can, collection(Item, sources, s => s.id)

[jvanbruegge]

Yeah, collection assumes sources.onion.state$ is an array$.

Does this API makes sense from a logical standpoint? How I understand it, I have to introduce a boilerplate component every time I want to use collection, just to fit the API. The problem is that it is not trivial to provide a mapped state stream to collection

You can, collection(Item, sources, s => s.id)

:+1:

[staltz]

Yeah, you're right about those downsides, I just don't know what to do about that.

The boilerplate component has a lot of smart logic to keep performance fast. And it's still possible to map the state stream, but you would do that outside, in another parent component.

[abaco]

Do we still want to allow keyless items in a list?

It would be nice. A quick solution would be to use a getKey function that computes a hash of the item's state. We could add such a function to the library, something like:

import {collection, getHash} from 'cycle-onionify';
...
const collectionSinks = collection(Item, sources, getHash);

[staltz]

I think all will be better once we have Immutable.js* support, because each data piece would have a hash function.

*: or some other library or solution that achieves the same goal, not necessarily Immutable.js.

[jvanbruegge]

maybe we can build upon #19

[staltz]

@abaco just for your information, I'm struggling to rewrite the examples/lenses to collection(), as it is. It's hard to determine the lens setter that takes the array of items and determines the currentIndex.

[staltz]

@jvanbruegge perhaps, yes. I'm interested in (1) allowing choice, (2) using or building a Proxy-based immutable JS library. ES6 Proxy isn't available in all major browsers, but building (2) is a good foundation for the future, and easier to migrate. I don't know if such library exists.

[abaco]

@staltz In the meanwhile we could just suggest users to use a hash function from an external library (e.g. object-hash. Something like a FAQ: "Do I really need to add keys to my list items? - You should, but if you're so lazy you can pass a hash function as getKey".

That examples/lenses might well be impossible to write with collection(). If that's the case it means list items lenses are more powerful than list lenses, which might be a good reason to add the lens argument to collection(). I don't know, I'll think about it.

[staltz]

@abaco I just wrote examples/lenses with collection(), https://github.com/staltz/cycle-onionify/commit/49a4e19b21526956876e926f167c8b3c3f493aa2 So it's possible. That said, it's a bit fragile because it's easy to put the program in an infinite loop if you create a new state object that has the same deep contents as the previous state object, because then strict equality won't work. This could be solved by using an immutable library.

[staltz]

I'll release a RC version of this so we can experiment with it elsewhere.

[staltz]

4.0.0-rc.1

[jvanbruegge]

If the API stays like this, I would include the boilerplate helper into onionify. So I can have a state object and do something like this:

function myComponent(sources) {
    const collectionSinks = isolateCollection(Item, sources, 'myArray', s => s.id);
}

isolateCollection is then basicly a wrapper around collection:

function isolateCollection(component, sources, scope, getKey = defaultGetKey) {
    const wrappedComponent = isolate(component, { onion: scope, '*': null });
    return collection(wrappedComponent, sources, getKey);
}

[abaco]

it's a bit fragile because it's easy to put the program in an infinite loop if you create a new state object that has the same deep contents as the previous state object

I can confirm that 😞.

I have a lens that creates a new object for each list item, so the state object that the item emits through the reducer is never what it will get through the source. Any update of the item's state triggers an infinite loop. Even the initial reducer does.

I should be able to restructure my state so that the identity of objects is kept throughout the lenses, but it's indeed quite fragile. It's not hard to have the library explode in your face by doing totally legitimate things. @staltz, do you think migrating to an immutable library is the only forward?

[staltz]

@staltz, do you think migrating to an immutable library is the only forward?

I wouldn't say it's the only way forward, but currently it's the only way I can imagine. Any other suggestions? I also really don't like the fragility. People will definitely hit this issue and they may not have the insight we are having.

[staltz]

I also noticed another issue that we need to solve: how to allow passing a name other than 'onion' to the collection. This feature was inconsistently supported, because collection() assumed the use of the name onion. Also, on the level of types, we were also assuming the name onion.

In a recent commit on the collection branch, I removed support for custom name, but just temporarily. The complication with the name field is that it exists in the StateSource, while collection() is just a helper function and has no access to the name.

I see two alternatives:

• Change the API to sources.onion.asCollection(Item, sources) so asCollection can have access to the 'name'
• Remove the custom name feature
  • This would mean onionify works by convention, and the name onion is magical. This goes a bit against Cycle's unmagical style, and could confuse people. On the other hand, in TypeScript, people would be guided by the types and it would be hard to do a typo unnoticed.
  • I also think other channels in Cycle.js like drivers allow names to be customized, so onionify would be a weird one

[staltz]

Actually I'm starting to like sources.onion.asCollection() for different reasons too. It reads a bit better and makes it more explicit the connection to sources.onion.state$. While collection(Item, sources) reads as "make a collection of Item using sources" (no reference made to sources.onion.state$ as an array$), sources.onion.asCollection(Item, sources) reads as "from the onion source, make a collection of Item using sources", which hints a bit better how does it make the collection: from the current onion source.

We could even consider expanding the semantics of sources.onion.asCollection to work when sources.onion.state$ is not a Stream<Array<T>> (currently it would crash, I guess). If sources.onion.state$ is something like Stream<object>, then sources.onion.asCollection(Item, sources) would create just one Item. I don't think there is practical utility for this, except it avoids one runtime crash and makes the API more resistant.

[jvanbruegge]

I like asCollection, but having Stream<object> create one item is useless. Better to be able to pass a lens that defaults to identity

[staltz]

It is useless indeed, but I think useless is better than error-prone, right? And we wouldn't document asCollection with Stream<object> in the docs.

[jvanbruegge]

The identity approach would have the same result, but a more useful side too

[staltz]

How do you envision that API?

[jvanbruegge]

sources.onion.asCollection(Item, sources, accessorLens = identityLens)

[abaco]

@staltz I'd be fine with sources.onion.asCollection(Item, sources, getKey = defaultGetKey).

@jvanbruegge If you add the accessorLens argument we end up with four arguments, because getKey is necessary - isn't it? I agree with André that four arguments is too many, especially when two are optional. However I understand that being forced to wrap a collection in a component is not the most convenient.

How would an equivalent to isolate, but for collections, look like? Maybe something like:

const instances$ = isolateCollection(sources.onion.asCollection(Item), 'list')(sources, s => s.id);

It would isolate the sources provided to the isolated collection, just like isolate would, but then isolate the sinks of each instance. .asCollection(Item) would need to return a curried function. What do you think?

[jvanbruegge]

i dont think 4 arguments is too much if two are optional. For the average user it is still a two argument function.

[staltz]

Hey guys, this is cool, and we are definitely making progress through discussions (reminder: we wouldn't even have lenses if it weren't for a healthy discussion between @abaco and me), but we're near to overengineering soon.

I think it's best when the API design is done following the principle of least surprise. It's not always easy to do the guesswork of how are beginners going to perceive the designed API, but some of this guessing can be done just through empathy.

So, how are beginners reading the readme.md going to perceive accessorLens or isolateCollection? What problem is it solving? Is the problem common or rare? I'm also not so sure what is the problem being targeted here. During this thread, I took note of some real problems to be solved, you can see the checklist in the first message in this thread.

[abaco]

Frankly I don't mind wrapping my collections in a component for the sole purpose of isolating the onion source/sink. I even put that component in a separate file. I actually like that.

But the main goal of components, as far as I see them, is not isolation, it's reusability. So I understand that Jan doesn't want to create a component that he won't ever reuse, just because he needs to isolate the collection. I think it's fair to expect that there's a way to directly isolate the collection, without any wrapper component. That's why I was thinking of a possible equivalent of isolate, but for collections not for components.

[abaco]

Alright, here is some working code (I've tried it):

function collectionScoped(scope: any): any {
  return function isolatedCollection(Item: any, sources: any, getKey: any) {
    const isolatedSources = {...sources, onion: isolateSource(sources.onion, scope)};
    const instances$ = collection(Item, isolatedSources, getKey);
    return instances$.map((instances: any) => {
      return {
        dict: instances.dict,
        arr: instances.arr.map((instance: any) => ({
          ...instance,
          onion: isolateSink(instance.onion, scope)
        }))
      };
    });
  }
}

You can use it this way, when your state is like {list: [1, 2, 3]}:

const instances$ = collectionScoped('list')(Item, sources);

Should we add this function to the library? How would that look like once we have sources.onion.asCollection instead of collection? Does it solve a common problem? I don't have answers. But collectionScoped does solve the problem of creating a collection from an array which is a property of the state object.

[staltz]

But collectionScoped does solve the problem of creating a collection from an array which is a property of the state object.

Am I understanding correctly that this is the problem? "We must have a way of creating a collection from list inside state = {a, b, c, list}, without introducing a List component"

[staltz]

As a different/orthogonal perspective to what I just said, maybe another problem is "how can beginners perceive that collection will use cycle/isolate under the hood", which is an issue with or without the proposed APIs of these last 5 or 6 messages.

[abaco]

"We must have a way of creating a collection from list inside state = {a, b, c, list}, without introducing a List component"

Exactly, that's how I understand @jvanbruegge's need.

[staltz]

So,

But the main goal of components, as far as I see them, is not isolation, it's reusability.

That's funny, because the need for a List component actually opens up more opportunities for reusability.

Notice how this component is quite generic, could be used in any situation where you need <ul>...<li>...</ul>:

function List(sources) {
  const instances$ = collection(Item, sources);

  const vdom$ = instances$
    .compose(pickCombine('DOM'))
    .map(vnodes => ul(vnodes));

  const reducer$ = instances$.compose(pickMerge('onion'));

  return {
    DOM: vdom$,
    onion: reducer$,
  };
}

[kylecordes]

@staltz and others, I've read this whole thread, and I agree very much with the notion that "taking over" sources.onion.state$ as then array$ is not wonderful. As far as I can tell, this means that the component can manage exactly one collection, it can't manage more than one collection, and it can't manage one collection plus some other onionified state. I can see the idea of getting around that by adding another layer of component around each collection needed, it seems as though that means more wiring for coordination among different parts of state for what would otherwise be part of the same component though.

Is there some way to shape both this collection mechanism, and other future helpers for managing contained components, in such a way that they can be composed and used side-by-side (one or more instances of each)?

[abaco]

@kylecordes, I think the source of this "not wonderful" situation is the fact that collection is a strange beast. It combines a few logically independent operations, it returns a strange stream of {dict, arr} that's only meant to be consumed by the specially designed helpers pickCombine and pickMerge. All this to improve performance. Staltz tried to slice collection into separate functions, but it seems impossible, precisely because efficiency requires carrying out several operations in conjunction. We traded some elegance for better performance - I think it was worth.

Now, given this strange beast collection, we should try to:

1. Keep it simple and don't add features unless strictly necessary. Selecting an array contained in the state object can be done out of collection, namely by isolateing a subcomponent. Hence the use of wrapper components.

2. Keep the weirdness of collection as localized as possible. collection, pickCombine and pickMerge are special tools that are meant to work in concert. They speak their own language (which includes {dict, arr}). Adding other helper functions that speak the same language (like the isolateCollection or collectionScoped I proposed) would spread the weirdness of collection around the app. That's not good. It's better to interface a collection with the rest of the app through a standard interface, i.e. the component. In this way we can use the standard tools that work with components, e.g. isolate.

I argued in a previous post that the main purpose of components is reusability, but that's wrong. The component is the central abstraction of Cycle: it describes the dataflow from sources to sinks from and to the external world, and a Cycle app is itself a component. Reusability is a consequence of the consistent adoption of this abstraction throughout Cycle (fractality, component utilities).

Also consider that the word component sounds important, but it's actually just a function. Stalz's List snippet above is a component, but it's so generic it can be considered a helper function. You could write a component factory that embeds a collection:

function makeCollection(ListItem, wrapper) {
  return function Collection(sources) {
    const instances$ = collection(ListItem, sources);
    return {
      DOM: instances$.compose(pickCombine('DOM')).map(wrapper),
      onion: instances$.compose(pickMerge('onion'))
    };
  }
}

const listSinks = isolate(makeCollection(Item, ul), 'list')(sources);

Or even define it inline:

const listSinks = isolate(sources => {
  const instances$ = collection(Item, sources);
  return {
    DOM: instances$.compose(pickCombine('DOM')).map(ul),
    onion: instances$.compose(pickMerge('onion'))
  };
}, 'list')(sources);

I think this sort of explicit code is the best way to deal with the weirdness of collection.

To answer your question:

Is there some way to shape both this collection mechanism, and other future helpers for managing contained components, in such a way that they can be composed and used side-by-side (one or more instances of each)?

I can't find a better way than using components: they are standard, lightweight, and fairly concise.

[staltz]

I absolutely agree with @abaco 's overview

[kylecordes]

Sounds good, I will continue working on an example here that coordinates two lists plus another bit of state. This is a mental adjustment versus frameworks/libraries which have more ceremony per component.

[staltz]

PS: next, I'll make the API sources.onion.asCollection and release that as a new RC. That would solve this problem mentioned in the first message

"How to keep the feature where we can pass a channel name instead of 'onion'"

And I just marked "How to avoid fragility to object equality issues?" as solved.

[staltz]

I just released v4.0.0-rc.7, and ticked these as solved

• How to keep the feature where we can pass a channel name instead of "onion"
• Do we still want to allow keyless items in a list?

We still allow basic array usage where you can isolate a child by an array index (see some examples in test.js), but I think whenever we want to support an actual list of many children that can be dynamically removed/added, we should support that only with the asCollection API.

Before releasing this version as non-RC, I need to rewrite more official examples, and we still have time to collect feedback.

[kylecordes]

@staltz rc.7 unfortunately has a debugger left in it:

https://github.com/staltz/cycle-onionify/commit/296c08b19f80d56aac006c8ba308b50181c47d64#diff-f41e9d04a45c83f3b6f6e630f10117feR151

[staltz]

Oops! Expect rc.8 very soon.

[kylecordes]

Thanks, using it now.

[staltz]

Idea! I'm building this just to cover a corner case " asCollection to support state$ when it's a stream of objects", but I also considered some objects like in Firebase or GunJS are a collection of stuff indexed by key. So if asCollection supports receiving an object Record<Key, Thing>, then it would create a collection of Thing. Then we would also need to modify pickCombine so that it returns a Stream<Record<Key, VNode>> instead of Stream<Array<VNode>>. But I'm not sure if it's worth doing this now, it could be just an improvement to happen in a version later on.

[abaco]

Why return Stream<Record<Key, VNode>> when you'll have to convert it to Stream<Array<VNode>> anyway, in order to feed it to the DOM driver?

What's the advantage of converting the output of pickCombine from Stream<Record<Key, VNode>> to Stream<Array<VNode>> compared to converting the input of asCollection from Record<Key, Thing> to Array<Thing>? It seems almost the same to me.

[staltz]

Yeah I guess you're right, the limiting factor is that snabbdom (DOM driver) expects children to be an array. Anyway this was just an idea for the future, we could come back to this later (if we want) in a different issue.