Custom generators (like `async`, `async*` and `sync*`)

[TekExplorer]

I think it could be really interesting if we could create our own generators like async, async*, and sync*

I noticed that really, a lot of the features present in each kind already exists in normal code, ala Completer, StreamController, and custom iterables

I looked at yield and yield* and equated it to StreamController.add and StreamController.addAll I recognized that async* returned an Iterable<Future<T>>, which Stream implements, not dissimilar to sync* I looked at await, and recognized how it flattened the callback hell of .thens by just giving us the value and early returning the Completer.future

I looked at these and realized they all return before a single line of code ever runs.

And I looked at #2567 https://github.com/flutter/flutter/issues/25280 and https://github.com/flutter/flutter/issues/51752 and remembered how @rrousselGit mentioned that "hooks" would be a language feature akin to doing final x = use thing;

So the feature that comes to mind is, what if we could make our own functional generators? extend the dart language "directly"?

What if we could do something like:

// Effect, Hook, whatever
// myGenerator could register disposals and such into the resulting object, to be handled elsewhere
Effect<...> makeThing() myGenerator { 
  // primitives would directly implement Effect (similar to flutter_hook's Hook class)
  final (get, set) = use state(0);
 // additional code

  return (get(), set);
}

// elsewhere:
// something like
final (value, setValue) = handler.use(makeThing()) // receives Effect<T>

// where:
T handler.use<T>(Effect<T> effect) {
  registerDispose(effect.dispose);
  // more logic
  return readEffect(effect);
}

and of course, that logic can be anything.

a naive example:

// pseudocode, may not be the best api
generator stringbuilder returns String {
  final buffer = StringBuffer();

  operator yield(String str) => buffer.write(str);
  operator yield*(Iterable<String> strs) => buffer.writeAll(str);

  String operator return() => buffer.toString();
}

Iterable<String> makeManyStrings(int count, String str) sync* {
  for (final i = 0, i < count; i++) {
    yield str;
  }
}
String makeString() stringbuilder {
  yield 'Hello ';
  yield 'World!';
  yield '\n';
  yield* makeManyStrings(3, 'Hehe');
}

final String result = makeString(); // Hello World!\nHeheHeheHehe

simple, but it shows how simple it could be, yet gives a ton of flexibility in terms of power and complexity isn't declarative code better?

• well okay, not always, but this can make code that would be better declarative, to be declarative

I think this can be a super powerful feature, allowing custom(ish) syntax to exist in packages (or sdks, hint hint) which could possibly simplify a lot.

It would make algebraic effects unnecessary since you'd have to provide them in the returned object before you can use any data.

the presence of yield would also provide "free" lazy loading for certain cases, like it does with stream and iterable

existing generators like async, async* and sync* could even be implemented in this feature, which would have the benefits of being able to look at its implementation (for better understanding and discovery) and also make it possible to add documentation to it.

Sometimes I try to hover over await or yield(*) to see what they would say and... nothing.

we can even reuse some existing syntax - await, yield etc would be definable operators

There's a lot of potential, and this would be a pretty major feature, so I'd like to hear some thoughts!

[hydro63]

I'm not really a fan of this feature, mostly because i don't really see a problem it solves, that a normal function doesn't. Features are not added just because they are interesting. Another problem i see, is that it feels very "meta", sort of like macro, but instead of generating boilerplate code, it changes how the entire language behaves. It's also quite difficult to read, and possibly ambiguous in some cases.

That being said, here are my suggestions: 1) Make it a class - there is no need to add a completely new construct to the language, instead just do the same thing macros did, and make it a special class

// small syntax change
generator class StringBuilder{
  final buffer = StringBuffer();

  operator yield(String str) => buffer.write(str);
  operator yield*(Iterable<String> strs) => buffer.writeAll(str);

  String operator return() => buffer.toString();
}

2) It's possible to make a general function class that would work as a framework on which to build a custom generator

// yield and yield* come from GeneratorFunction
function class StringBuilder<T> implements GeneratorFunction {
  final buffer = StringBuffer();

  @override
  operator yield(String str) => buffer.write(str);
  @override
  operator yield*(Iterable<String> strs) => buffer.writeAll(str);

  @override
  String operator return(T retValue) => buffer.toString();
}

And the reason i would do this is because of it giving it a greater use case, for example having a go-like defer, or to wrap the result.

function class DeferredStatement<T> implements GeneratorFunction {
  List<Function> deferredFuncs = [];
  @override
  operator yield(void Function() func) =>deferredFuncs.add(func);

  @override
  T operator return(T retValue){
    for(final func in deferredFuncs){
      func();
    }
    return retValue;
  }
}

String doSomething() DeferredStatement {
  // i KNOW that in Dart the files are automatically closed, but it's just demonstration
  File f = File("text.txt");
  yield () => f.close();
  return f.readAsStringSync(); //after return runs the yielded function
}

It's also possible to wrap the result in somehting like this

// CatcherFunction -> interface for catching exceptions that occured in function
function class ResultWrapper<T> implements CatcherFunction {
  @override
  Result<T> catch(T err) => Failure(err);

  @override
  Result<T> operator return(T retValue) => Success(retValue);
}

String toThrowOrNotToThrow() ResultWrapper {
  String out = funcCanThrow(); // if throws -> Failure  
  return out;  // success
} // actual return type is Result<String> bcs of ResultWrapper

Possible problems of my proposal, is that it obfuscates the return type of the function, which is not ideal. A possible solution is to have the returns hint that was in your proposed syntax.

That being said, it still is very meta, so i'm not really sure if it's good to have this before we have macros. Also, for such a big change in the language (both yours, and my proposed syntaxes), extensive discussion is needed to make it work. As such, @lrhn, could you please provide feedback.

My standpoint on either of the proposals is that it would be good to have, but the benefit of having this is not really proportional to the difficulty of implementing it correctly

[lrhn]

This sounds like monads, which means it's probably something that won't work well with the Dart type system if you try to abstract over it. That requires a higher-order type system which Dart doesn't have.

If you don't try to generalize, then it's more like normal functional programming.

String stringBuilder(
  void Function({required void Function(String) yield, required void Function(Iterable<String>) yieldAll}) body) {
  var buffer = StringBuffer();
  body(yield: buffer.write, yieldAll: buffer.writeAll);
  return buffer.toString();
}

String writer(Input input) => stringBuilder(({yield, yieldAll}) {
  while (imput.something) {
      yield(input.value);
  }
  yieldAll(input.rest);
});

You can even abstract over writers:

typedef Writer<R, T> = R Function(
    void Function({required void Function(T) yield, required void Function(Iterable<T>) yieldAll}));

but not over "body modifiers" in general, which means that it's likely not a good general language feature.