So many implementations :)

[vitaly-t]

It is interesting, how many implementations I keep finding, after having done one of my own - iter-ops. My implementation is without use of any synthetic types though.

Just wanted to ping you for sharing experience in writing such libraries :smile:

[Yomguithereal]

Hello @vitaly-t,

My implementation is without use of any synthetic types though.

What do you mean by synthetic types in this context?

Just wanted to ping you for sharing experience in writing such libraries

Do you have specific topics in mind? From my end obliterator is mostly backing mnemonist and graphology and I tend to use it mainly to 1. harmonize the maaaaany JavaScript iteration schemes and 2. provide some ways of doing some things in a streaming fashion to avoid wasting memory. Because unfortunately, iterators and generators remain poorly optimized by V8 and Gecko and I cannot always rely on them in performance contexts.

[vitaly-t]

What do you mean by synthetic types in this context?

By synthetic types, I mean any type that wraps iterables to provide operands from that type. In my library, I am not wrapping anything up at all, I'm working with iterables directly, from one operator to the next. This makes it possible to intercept the processing in any way you want, simplify creation of custom operators, as well as integration, because there is no entry point with a starting synthetic type, you just pick up iterable where it needs processing.

Do you have specific topics in mind?

Not specifically, I have been working on it just recently, that's why it peaked my interest. I still keep discovering how many implementations out there.

iterators and generators remain poorly optimized by V8

I would agree about the generators, but iterators are just native JavaScript, and it is as good as you implement it.

[Yomguithereal]

By synthetic types, I mean any type that wraps iterables to provide operands from that type. In my library, I am not wrapping anything up at all, I'm working with iterables directly, from one operator to the next. This makes it possible to intercept the processing in any way you want, simplify creation of custom operators, as well as integration, because there is no entry point with a starting synthetic type, you just pick up iterable where it needs processing.

Mine does not use synthetic types either, per your definition. I only use an Iterator class which is basically sugar to create a plain JavaScript iterable iterator: https://github.com/Yomguithereal/obliterator/blob/master/iterator.js

Not specifically, I have been working on it just recently, that's why it peaked my interest. I still keep discovering how many implementations out there.

Yes I would expect this is a common use case :). I am not expecting people to use this library but as anything I code it is Open Source and used throughout my lab's works. What's more, since it is used by many of our own libs, I need to be able to keep some control over this particular code and this is the reason why I did not use a third-party library for this (as many exist as you could see :) ) and implemented stuff to fit my specific use-cases.

I would agree about the generators, but iterators are just native JavaScript, and it is as good as you implement it.

What I meant is that you cannot expect a JavaScript iterator to be as fast a custom #.forEach function nor as fast as array or object iteration. Which means that you should probably avoid it if what you need is to be as performant as possible (for very low-level & hot code). But I personally enjoy them very much for application code etc. I am a big fan of python iterator/generator protocols and I just wished it was this easy in JavaScript also without requiring quirky heuristics such as this one to ensure you can iterate over pretty much anything.

[vitaly-t]

What I meant is that you cannot expect a JavaScript iterator to be as fast a custom #.forEach function nor as fast as array or object iteration

Actually, you can. Just as you start chaining iterators, and merging multiple operators, it becomes faster than standard Array functions, due to a single iteration to do all processing, which is far more efficient.

I have done some performance measurements of Array.from().filter().map() versus my pipe(filter(), map()), and the latter was 2.5 times faster.

[vitaly-t]

May I ask you why you thought that #26 was an issue?

Type IteratorResult clearly puts done as optional.

Just curious, because I chose not to provide done: false, because it is useless.

[Yomguithereal]

Actually, you can. Just as you start chaining iterators, and merging multiple operators, it becomes faster than standard Array functions, due to a single iteration to do all processing, which is far more efficient.

You can also just do your operations in the body of the loop and it will be way faster. I am not denying that if you need an abstraction like filter or map it could not be faster than Array's one. But not if you consider doing the operations themselves within the loop.

var v;
for (var i = 0; i < a.length; i++) {
  v = mapper1(a[i]);
  v = mapper2(v);
  if (!filter1(v)) continue;
  // etc.
}

^ this will always be faster than relying on iterators, or array methods.

[Yomguithereal]

Just curious, because I chose not to provide done: false, because it is useless.

I include a {done: false} because I observed some deopt on v8 due to the required casting of undefined => bool in some scenarios. The same sometimes happen in Gecko also. By ensuring I have done as a true boolean I sometimes avoid JS engines tripping themselves over.

[vitaly-t]

By ensuring I have done as a true boolean I sometimes avoid JS engines tripping themselves over.

Did you mean I have done as a false boolean? The true value has no question about, it is the false that I'm not sure why it is ever needed, especially when even NodeJS TypeScript declarations clearly state it is optional.

As for the loop example you showed - sure, it can be optimized. But it can be very time-consuming, and too inefficient these days. And just as we throw in something more complex, like distinct() operator, it will get messy :)

[Yomguithereal]

Did you mean I have done as a false boolean

No I mean that step.done is now always boolean in my case, not undefined, and I guarantee it. Usually engines optimize for the case when an undefined value is tested (being evidently falsey) but sometimes their runtime analysis fail and it will need to cast those undefined to a true bool when performing the test over the value and then you lose the cost of this casting operation. But this is quite rare and if you are able to prove to me that removing the done property altogether actually increases performance in most case I would strongly reconsider my #26 decision.

But it can be very time-consuming, and too inefficient these days. And just as we throw in something more complex, like distinct() operator, it will get messy :)

Yes, and this is why we use abstractions. But in some cases it is worthwhile not to use those if what you need is to have the most performant code possible. In 99% of the use-cases you don't. But if what you need is to ensure the iteration function of a lru cache called millions of times per minutes is as fast as possible, then it becomes relevant.

This is what I meant by "Which means that you should probably avoid it if what you need is to be as performant as possible" that is to say I do not use iterators when developing some code that absolutely requires to be as fast as it can be (such as the code in mnemonist & graphology). If I don't care about performance, I will gladly use iterators and even for ... of loops (that are demonstrably slower than plain iterators, used with operations as you do) because they are way clearer and easily composable.

This said I also agree with you when you say "iterators are just native JavaScript, and it is as good as you implement it." And I hope I did a good job of this in this library. But I am sure we can find ways to optimize both our codes by reading each other's one.

[Yomguithereal]

For instance, here you avoid using recursion by using a do...while construct. This is clearly more efficient than what I did and avoids stack overflow on some large succesions of cases when the predicates returns false, I think. I usually use loops & stacks to avoid recursion but here it seems I forgot.

[vitaly-t]

Yes, I fully stayed away from any recursions, those are horrible. Since you quoted code in my of my operators, I will tell you that the most challenging and interesting was concat operator, took awhile to make it work right in terms of the logic + TypeScript type inference.

[Yomguithereal]

Yes, I always hate when requiring something that starts looking like a state machine to implement a custom iterator. I have an horrible example of this here.

Your code here won't accept iterators then? Only iterables? Not saying it should but you never stumbled some code that was not implementing the iterable protocol, only the iterator one (that you can only duck-type ofc...)?

[vitaly-t]

Your code here won't accept iterators then? Only iterables?

Yes, I implemented it consistent with the type above that lists what's accepted:

/**
 * Value or Iterable
 */
type VI<T> = T | Iterable<T>;

I can surely extend it to support iterators also, it would be very easy, in fact, it didn't occur to me the first time. After all, I finished it all just recently. I'm sure some improvements will find their way in :)

[vitaly-t]

b.t.w. I have already released an update where concat supports iterators :wink: Thanks for spotting that!

[Yomguithereal]

As do I to avoid recursion in filter 😉

[vitaly-t]

I've added a few new operators since yesterday - last, isEmpty and defaultEmpty. But you probably aren't interested in that anymore, so feel free closing this issue then 😉

[Yomguithereal]

I'll close the issue then @vitaly-t. Feel free to reopen it you want to discuss anything or chat :)