Generator functions

[RX14]

A way to transform (at compile time) a yielding function into an iterator would make writing iterators vastly simpler and easier to read. A good usecase would be allowing the iterator-returning and yielding functions of Enumerable to share code, reducing bugs and maintenance effort. An existing implementation of this concept (for ES2015 generators) is available here.

See: https://github.com/crystal-lang/crystal/issues/4198 and https://github.com/crystal-lang/crystal/issues/3478.

[asterite]

If compile times are already slow and consume a lot of memory, I can only imagine what limits the compiler will be reaching with this :-)

This was considered in the past, but blocks makes it very complex to do. If someone wants to implement it and send a PR, please go ahead.

[RX14]

@asterite This seems to me to be quite a local and scoped transform though, I wouldn't have thought it would add much compared to the global typing passes which touch the whole program.

That being said, it is a hard transform to get right, especially with blocks, so this is a far-future thing, we have more important things to do right now.

[faultyserver]

A good usecase would be allowing the iterator-returning and yielding functions of Enumerable to share code, reducing bugs and maintenance effort.

This sounds to me a bit like Ruby's enum_for and to_enum. SO gives a good usage example.

I could see that being useful at runtime, but I'm not exactly sure how it could be done at compile time...

EDIT: Reading through the linked issues, I see how this could be done at compile time. Would those iterator and yielding functions then be merged to one function that can do both? Or would they just share some code but remain separate?

[bew]

FWIW I did a naive proof-of-concept implementation of a generator: https://carc.in/#/r/4gi7 It's not optimal at all, as it relies on fibers/channels and it doesn't cleanup the fiber when the iterator didn't finish and is GC'ed..

I did a generator with automatic type inference (abusing typeof, based on the implementation of #4813), and one where you have to specify the types (I personally prefer this latter one).

They can be used like this:

def each_dir
  yield "dir1"
  yield "dir2"
end

# automatic type inference
it = generator each_dir
p it.to_a # => ["dir1", "dir2"]

# given type
it = generator each_dir, String
p it.to_a # => ["dir1", "dir2"]

There is also a generator for when the method yields multiple arguments:

def each_multi_args
  yield 41, false
  yield 42, true
end

# given type
it = generator_multi_args each_multi_args, {Int32, Bool}
p it.to_a # => [{41, false}, {42, true}]

# was too lazy to do the automatic type inference one ^^

---

Implementation:

# Used in the 'typeof' of the automatic type inference' generator
private struct TypeSentinel
end

# Generator (single block arg) with automatic type inference
macro generator(expr)
  %chan = Channel(Iterator::Stop | typeof(begin
    %item = TypeSentinel.new
    {{ expr }} do |x|
      %item = x
    end
    raise "" if %item.is_a? TypeSentinel
    %item
  end)).new

  spawn do
    {{ expr }} do |x|
      %chan.send x
    end
    %chan.send Iterator.stop
  end

  Iterator.of { %chan.receive }
end

# Generator (single block arg) with specified type
macro generator(expr, type)
  %chan = Channel(Iterator::Stop | {{ type }}).new

  spawn do
    {{ expr }} do |x|
      %chan.send x
    end
    %chan.send Iterator.stop
  end

  Iterator.of { %chan.receive }
end

# Generator (multiple block args) with specified tuple type
macro generator_multi_args(expr, type)
  %chan = Channel(Iterator::Stop | {{ type }}).new

  spawn do
    {{ expr }} do |*args|
      %chan.send args
    end
    %chan.send Iterator.stop
  end

  Iterator.of { %chan.receive }
end

WDYT?