No way to do ES6 'for .. of' loops for generators?

[jagill]

Given a generator

gen = ->
  j = 0
  while j < 9
    yield j++
  return

how do we loop through it via the javascript for .. of loops? In javascript, we'd do

for (i of gen()) {
  console.log(i);
}

The coffeescript for i in gen() gives the C-style for (_i = 0; _i < gen().length; i++) loops, and for i of gen() gives for (i in gen()). Is there some way to get for (i of gen()) that I didn't see in the documentation?

[ssboisen]

I came to ask the same question. Having to manually run next and check for done is quite counter to the syntax-goodness that CS normally prides itself of.

[jashkenas]

Shoot. That's a major bummer.

@alubbe et. al — any ideas here? Do all of the runtimes that support generators also support ES2015's for of?

[alubbe]

@jashkenas to my knowledge only chrome, firefox and opera support generators and all three support for .. of - so the answer would seem to be yes.

Syntax wise, this will be a bit of a challenge. for all .. of/in? each ... of/in?

[michaelficarra]

We might have to add a flag for this to compile CoffeeScript for-in to ES6 for-of. I don't see any better way.

[jagill]

@michaelficarra That sound problematic -- a given script could either loop through generators, or arrays.

Node 0.11 and iojs also support for of.

I'd much rather have a special syntax, like for gen x of myGen().

[michaelficarra]

@jagill: ES6 for-of works with arrays because they are iterables.

[epidemian]

I think @michaelficarra's suggestion is the most sensible approach.

I personally would prefer releasing a backward incompatible version of CS (2.0, 3.0, what have you) that aligned for in and for of semantics with JS and have a conversion tool that can be run reliably and transform CS code from one version to the other. After all, it's just a deterministic syntax transformation.

I'm not in favour of adding yet another looping construct to CS.

[vendethiel]

if they're supposed to be used by generators, can't we use something like the currently-invalid from x from y()?

[lydell]

@vendethiel did you mean for x from y()? I like that.

A current workaround would be something like:

forOf = (gen, fn) ->
  `for (value of gen()) fn(value)`

forOf gen, (i) -> console.log i

[vendethiel]

Yes, sorry.

[alubbe]

I dislike creating type-specific loop declarations. The very power of for ... of lies in looping through all iterables: arrays, (invoked) generators, sets, maps and, interestingly, arguments.

So fundamentally, there are two routes and we'll need @jashkenas to give us a pointer: 1) Backwards-compatible: will need a new syntax like each ... of 2) Breaking compatibility: fundamentally rethinking how for ... in and for ... of work in coffee-script. @michaelficarra and @epidemian have offered two ways this could be done

[ssboisen]

Couldn’t the same syntax be used? If the context of the for is a generator it compiles to for of otherwise we compile using the current strategy. We can assume the for of feature is available if users use yield and the required information about weather the context is a generator should also be available?

[Artazor]

@ssboisen, the problem is that we can't determine the context of the for at the compile time (imagine it comes from a function parameter). So we can't choose the right way to compile. Runtime check will have significant performance hit.

@jashkenas, I'd vote for backward incompatible CS 2.0 (according to semver!) If we'll introduce a new looping construct (to maintain backward compatibility) then it will replace the original in eventually in everyday usage, and we will have an obsolete syntax: why to use in if we can use ??? for looping over iterables polymorphicaly and cheaply?

Breaking changes will allow introduce new keywords for existing problems as well (I mean await prefix operator for #3813 and #3757)

[jashkenas]

why to use in if we can use ??? for looping over iterables polymorphicaly and cheaply?

Ha, ha, suckers! :wink:

Never assume that a new feature in JS is going to be "cheap" or smart to use — not when its freshly implemented, and probably not even after many years of use.

Check this out. Run it yourself.

http://jsperf.com/for-in-vs-for-of

[alubbe]

nice! that is some next level performance if I've ever seen one.

@jashkenas can you help me interpret whether your response means that your are leaning towards a new third syntax or changing CS's loops? :)

[jashkenas]

I don't know of a great solution just yet — but I haven't really had a chance to sit down and give it a think. There are, however, ground rules:

• We're not adding flags to CoffeeScript.
• It would be better to revert yield and remove generators for now, than to have to add a flag.
• We're not going to hurt regular array iteration performance to support this new feature.
• It's undesirable (but perhaps not unavoidable) to introduce a new looping form.
• Since for of performance is so shitty — maybe we can compile down to something else and still beat it handily?
• I wish we had considered this wrinkle before merging the yield PR ;(

[alubbe]

Regarding 2: please please please no! for ... of is a feature independent of generators and should have no impact on whether we keep this new awesome feature around or not. It is a loop over iterables, and an instantiated generator just happens to be iterable. Regarding 6: (see 2) Also, it's awesome because finally more people are joining the discussion, reporting bugs and actually writing better code than before.

[alubbe]

Here is more information on what for ... of does: https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/The_Iterator_protocol

Basically, it converts strings, arrays, maps, etc. into a generator, instantiates it and continually calls .next() on it. I assume that is where all of the performance is lost.

Looking at it, I think CS could go without for ... of for the time being. It's quite slow and, if you really ever need for x of y, you can use

_ref = y[Symbol.iterator]()
while _next = _ref.next(), val = _next.value, !_next.done
  ...

[jashkenas]

That's sort of the kind of thing I'm talking about ... Is a direct:

_ref = y[Symbol.iterator]()
while x = _ref.next()
  ...

... a lot faster than a for of call? If it is — should we have a construct or a looping hint that compiles into it?

[alubbe]

No, it is just as slow: http://jsperf.com/for-in-vs-for-of/2

[vendethiel]

@jashkenas you don't want "for x from y" as a third syntax then?

[jashkenas]

Apropos of nothing — what the hell is up with the:

generator = array[Symbol.iterator]()

... bullshit? Are we trying to pretend like we're not in a dynamic language that embraces duck typing any more? On what planet would that be preferable to:

generator = array.iterator()

Maybe for ES2019, we'll need a new third type of strings, after strings and symbols, to avoid name clashes again, so that we can have:

newFancyIterator = array[Symbol2019.iterator]()

<\/grump>

[jagill]

One of the things I've liked about CoffeeScript is that it allows me to do anything I could do in Javascript, modulo a couple bad constructs. Assuming we don't think the for of loop for generators is a bad construct, IMO CS should support it (or an equivalent), at least someday. It also looks like there's no possibility any time soon to use the same syntax for normal array loops and for of. So I don't see an alternative to a third syntax.

That being said, I have less absolute feelings about when it happens. For my own CS use (since iojs and "soon" Node 0.12 support generators), I'd prefer it to be sooner rather than later.

[epidemian]

Apropos of nothing — what the hell is up with the:

generator = array[Symbol.iterator]()

... bullshit? Are we trying to pretend like we're not in a dynamic language that embraces duck typing any more

Yeah, it's lamentable. But the philosophy of never breaking backwards compatibility, nor breaking existing applications, plus the common practice of extending native objects, makes it impossible to extend the language in a sane way like adding a normal iterator method to arrays.

Hell, not even something as simple as Array#contains can be added to the standard library because some library monkey-patched it and there are tons of applications relying on it being something else by now.

[devongovett]

I think for x from gen (or another syntax) is the only way that really makes sense here, unfortunately. Otherwise we basically have to either know the type in the compiler (not currently possible), or do a runtime check to see if its an object or iterator, which will be slow.

[lydell]

ES5 has two for loops. So does CS. CS for-in is ES5 for with shorter syntax. CS for-of is ES5 for-in. ES6 added a third for loop. Why can't CS too? CS for-from would be ES6 for-of.

[jashkenas]

But wouldn't it just be super tragic for:

CoffeeScript	ES6
for-in	for
for-of	for-in
for-from	for-of

... especially when, IIRC, ES6's for-of was partially a syntax "cowpaved path" borrowed from CoffeeScript?

It's even more tragic when you consider that in a perfect language, you wouldn't have three different syntaxes for these loops — you would only have one that handled arrays, objects and iterables.

[devongovett]

Yeah, it sucks and there should definitely be one loop type ideally. But unless you want to implement type-inference in the compiler (hard), or type checks at runtime (slow), we're kinda stuck right?

[edgebal]

My two cents: What about adding a keyword to for...of to let the interpreter know it's a generator?

for i of yielded gen()
  code...

• or -

for i in yielded gen()
  code...

One of those would compile to for (i of gen()) { code... }

Cheers!

[baslr]

my idea:

gen = () ->
  index = 0

  while index++ < 10
    yield index

a = gen()

for val next a
  console.log val

because An object is an iterator when it implements a next()

[baslr]

@lydell

forOf = (gen, fn) ->
  `for(value of gen()) fn(value)`
  undefined

otherwise

    return for(value of gen()) fn(value);
           ^^^
SyntaxError: Unexpected token for

[pflannery]

Haha yep for-of performance sucks. Its purposely bailed out in v8 so it's performance is bad. But we have to bear in mind it's still a draft spec and things could change. Hopefully it will be much better when the first spec is finalised.

I think if there is a new loop syntax introduced it should be pronounceable in English.

[dyoder]

+1 @alubbe re: yield being a separate and awesome feature, which i've been using like crazy since it became available, without ever need iterator support. (Thank you.)

Key point: people keep introducing generator-specific syntax, when the issue is iterator support. generators just happen to be iterators. that's the only connection. We want iterator support here.

Since current for ... in performance is far better than using iterators, we're not going to slow it down to check for an iterable, and we don't want a third syntax, that means this isn't a coffeescript feature for the foreseeable future, right?

FWIW, you could generate a cheap runtime check to see if you're dealing with an iterator by checking for length or next, since these must be defined anyway. This does not appear to be appreciably slower than the current compiled code, at least on modern browsers:

http://jsperf.com/for-in-vs-for-of/6

[sebmck]

You could always do something like:

for (var i of nums) {}

for (var _iterator = nums, _isArray = Array.isArray(_iterator), _i = 0, _iterator = _isArray ? _iterator : _iterator[Symbol.iterator]();;) {
  var _ref;

  if (_isArray) {
    if (_i >= _iterator.length) break;
    _ref = _iterator[_i++];
  } else {
    _i = _iterator.next();
    if (_i.done) break;
    _ref = _i.value;
  }

  var i = _ref;
}

Produces much more code but it puts arrays in the fast-path while also supporting all iterables.

[MetaMemoryT]

@sebmck , your solution must run an if branch statement every step of the loop.

how about this CoffeeScript:

for e of iterator
  <code>

compiles into this ES6 javascript:

if (Array.isArray(iterator)) {
  for (var i = 0, len = iterator.length; i < len; i++) {
    var e = iterator[i];
    <code>
  }
} else {
  for (var e of iterator) {
    <code>
  }
}

Pros:

• easy to implement
• easy to transition once browsers have VM optimization on Array iterator loops.
• until the optimization is made:
  • the extra js being generated is not that expensive when it is gzipped

Cons:

• until the optimization is made:
  • code in for loops will be generated twice
  • 1 isArray method call is ran at run-time at the beginning of each loop.

CoffeeScript for loops with 2 variables:

for key, value of object
  <code>

can remain the same.

[sebmck]

@MetaMemoryT Yes and the overhead of that if is extremely minimal. Copying the entire loop body is not at all practical.

[pikeas]

So what's the current best way to step through a generator in CS?

[luochen1990]

Actually, I think a function called foreach will be enough. since es6 for..of is already slow enough :) ie. you can enumerate a range like this:

foreach range(5), (x) ->
    console.log x

this foreach is implemented here and you can try it here

[babakness]

@MetaMemoryT +1 I like this since we also consider that when the key,value of obj and syntax is in place, we don't need this check.

Only in the case of item of mysteryThing do we need to check if we are dealing with an obj, array or generator. My 2 cents is that the syntax of key of obj be depreciated but supported for a little while before the behavior is consistent with ES6. Besides, the current key of obj is reverting to key in obj which is kind of strange anyway in my opinion.

[bernhard-42]

Some further thoughts:

In ES2015 I understand there are at least three ways of iterating over a generator (see https://leanpub.com/exploring-es6/read, 21.3.1):

• for ... of (broken in coffeescript)
• let [x,y] = gen() (desctructuring, doesn't seem to work in coffeescript right now)
• let arr = [...genFunc()]; (spread operator, doesn't seem to work in coffeescript right now)

What is the idea to support in coffeescript?

Since I anyhow have to call coffee --nodejs --harmony_generators test.coffee because ES5 doesn't support function*, why not all of them since the underlying transpiled javascript is anyhow not ES5?

I guess the problem is to distinguish between a generator object and "normal" object for for...to and the assignment operator = to determine whether a ES5 or an ES6 construct is necessary.

Maybe typeof obj[Symbol.iterator] == "function" (again taken from https://leanpub.com/exploring-es6/read) could help? To take the example of for...of:

if (typeof obj[Symbol.iterator] == "function") {
    // use ES6 for...of
} else {
    // use ES5 for...in
}

[lydell]

let [x,y] = gen() (desctructuring, doesn't seem to work in coffeescript right now)

Are you saying that isn’t equivalent to the following?

let tmp = gen()
let x = tmp[0]
let y = tmp[1]

[bernhard-42]

no, I mean destructuring of generators. While it should work in ES6/ES2015, in coffeescript

squares = ->
    num = 0
    while num < 2
        num += 1
        yield num * num
    return

[x, y] = squares()
console.log x, y

returns undefined undefined

[lydell]

$ coffee -bpe '[x, y] = squares()'
var ref, x, y;

ref = squares(), x = ref[0], y = ref[1];

So you mean that the above is equivalent to var [x, y] = squares() but still does not work as intended?

[bernhard-42]

maybe I miss something, so here is my terminal output

$ cat test2.coffee
squares = ->
  num = 0
  while num < 2
    num += 1
    yield num * num
  return

[x, y] = squares()
console.log x, y

$ coffee --nodejs --harmony_generators test2.coffee
undefined undefined

$ coffee -bp test2.coffee
var ref, squares, x, y;

squares = function*() {
  var num;
  num = 0;
  while (num < 2) {
    num += 1;
    (yield num * num);
  }
};

ref = squares(), x = ref[0], y = ref[1];

console.log(x, y);

I would not have expected undefined undefined

[bernhard-42]

please see also https://github.com/jashkenas/coffeescript/issues/4018 for some surprising generator behaviour (at least for me)

[michaelficarra]

Yes, @lydell, it is different. It does not use indexing internally, but instead calls next on the iterable repeatedly.

[lydell]

@michaelficarra Thanks for explaining!

[bernhard-42]

@michaelficarra So the only way to access elements of a generator in coffescript right now is to iterate over it using .next(), e.g.

iterator = squares()
until ((it = iterator.next()).done)
  doSomething it.value

No for loops and no deconstruction, right?

[michaelficarra]

@bernhard-42 As of right now, yes.

[bernhard-42]

@michaelficarra Thanks