Static Metaprogramming

[jakemac53]

Metaprogramming refers to code that operates on other code as if it were data. It can take code in as parameters, reflect over it, inspect it, create it, modify it, and return it. Static metaprogramming means doing that work at compile-time, and typically modifying or adding to the program based on that work.

Today it is possible to do static metaprogramming completely outside of the language - using packages such as build_runner to generate code and using the analyzer apis for introspection. These separate tools however are not well integrated into the compilers or tools, and it adds a lot of complexity where this is done. It also tends to be slower than an integrated solution because it can't share any work with the compiler.

Sample Use Case - Data Classes

The most requested open language issue is to add data classes. A data class is essentially a regular Dart class that comes with an automatically provided constructor and implementations of ==, hashCode, and copyWith() (called copy() in Kotlin) methods based on the fields the user declares in the class.

The reason this is a language feature request is because there’s no way for a Dart library or framework to add data classes as a reusable mechanism. Again, this is because there isn’t any easily available abstraction that lets a Dart user express “given this set of fields, add these methods to the class”. The copyWith() method is particularly challenging because it’s not just the body of that method that depends on the surrounding class’s fields. The parameter list itself does too.

We could add data classes to the language, but that only satisfies users who want a nice syntax for that specific set of policies. What happens when users instead want a nice notation for classes that are deeply immutable, dependency-injected, observable, or differentiable? Sufficiently powerful static metaprogramming could let users define these policies in reusable abstractions and keep the slower-moving Dart language out of the fast-moving methodology business.

Design

See this intro doc for the general design direction we are exploring right now.

[rrousselGit]

Would static function composition be in the scope of this feature?

At the moment, higher-order functions in Dart are fairly limited since they require knowing the full prototype of the decorated function.

An example would be a debounce utility:

void Function() debouce(Duration duration, void Function() decorated) {
  Timer? timer;
  return () {
    timer?.cancel();
    timer = Timer(duration, () => decorated());
  };
}

which allows us to, instead of:

class Example {
  void doSomething() {

  }
}

write:

class Example {
  final doSomething = debounce(Duration(seconds: 1), () {

   });
}

but that comes with a few drawbacks:

• obvious readability decrease
• our debounce utility is not very reusable. It works only on void Function(), but we'd like it to work for all functions.

With static meta-programming, our debounce could inject code in the class at compilation, such that we could write:

class Example {
  @Debounce(Duration(seconds: 1))
  void doSomething() {
    print('doSomething');
  }

  @Debounce(Duration(seconds: 1))
  void doSomethingElse(int value, {String named}) {
    print('doSomethingElse $value named: $named');
  }
}

[jakemac53]

There is a delicate balance re: static function composition, but there are certainly many useful things that could be done with it. I think ultimately it is something we would like to support as long as we can make it obvious enough that this wrapping is happening.

The specific balance would be around user confusion - we have a guiding principle that we don't want to allow overwriting of code in order to ensure that programs keep their original meaning. There are a lot of useful things you could do by simply wrapping a function in some other function (some additional ones might include uniform exception handling, analytics reporting, argument validation, etc). Most of these things would not change the meaning really of the original function, but the code is being "changed" in some sense by being wrapped.

Ultimately my sense is this is something we should try to support though. I think the usefulness probably outweighs the potential for doing weird things.

[mateusfccp]

I like Lisp approach (in my opinion, the utmost language when it comes to meta-programming). Instead of defining a @Debounce or something alike, we would define new syntax that would simply expand to a regular method at compile-time. I don't know, however, how much complex is to make something like this considering Dart syntax.

[lrhn]

For something like debounce, a more aspect-like approach seems preferable. Say, if you could declaratively wrap a function body with some template code:

class Example {
  void doSomething() with debounce(Duration(seconds: 1)) {
    print('doSomething');
  }

  void doSomethingElse(int value, {String named}) with debounce(Duration(seconds: 1)) {
    print('doSomethingElse $value named: $named');
  }
}

template debounce<R>(Duration duration) on R Function {
  template final Stopwatch? sw;
  template late R result;
  if (sw != null && sw.elapsed < duration) {
    return result;
  } else {
    (sw ??= Stopwatch()..start()).reset();
    return result = super;
  }
}

This defines a "function template" (really, a kind of function mixin) which can be applied to other functions. It cannot change the signature of the function, but it can access arguments (by forwarding them as with templateName(arg)), and it can do things before and after the original body is run. The template variables are per-template instantiation variables (just as we could declare static variables inside normal functions).

(Maybe we just need AspectD for Dart.)

[rrousselGit]

It cannot change the signature of the function, but it can access arguments

But an important part of function composition is also the ability to inject parameters and ask for more parameters.

For example, a good candidate is functional stateless-widgets, to add a key parameter to the prototype and inject a context parameter. This means the user would define:

@statelessWidget
Widget example(BuildContext context, {required String name}) {
  return Text(name);
}

and the resulting prototype after composition would be:

Widget Function({Key? key, required String name})

where the final code would be:

class _Example extends StatelessWidget {
  Example({Key? key, required String name}): super(key: key);

  final String name;

  @override
  Widget build(BuildContext) => originalExampleFunction(context, name: name);
}

Widget example({Key? key, required String name}) {
  return _Example(key: key, name: name);
}

[jakemac53]

I definitely agree we don't want to allow for changing the signature of the function from what was written. I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want. The original function might be private in that case.

[rrousselGit]

I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want

That's what functional_widget does, but the consequence is that the developer experience is pretty bad.

A major issue is that it breaks the "go to definition" functionality because instead of being redirected to their function, users are redirected to the generated code

It also causes a lot of confusion around naming. Because it's common to want to have control on whether the generated class/function is public or private, but the original function to always be private.

By modifying the prototype instead, this gives more control to users over the name of the generated functions.

[jakemac53]

Allowing the signature to be modified has a lot of disadvantages as well. I think its probably worse to see a function which is written to have a totally different signature than it actually has, than to be navigated to a generated function (which you can then follow through to the real one). You can potentially blackbox those functions in the debugger as well so it skips right to the real one if you are stepping through.

[bouraine]

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

[mateusfccp]

@bouraine

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

Yes.

[jakemac53]

@tatumizer This issue is just for the general problem of static metaprogramming. What you describe would be one possible solution to it, although we are trying to avoid exposing a full AST api because that can make it hard to evolve the language in the future. See https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md for an intro into the general design direction we are thinking of here which I think is not necessarily so far off from what you describe (although the mechanics are different).

[idkq]

Great intro & docs.

Hopefully we'll stay (far far) away from annotations to develop/work with static meta programming?!

[jakemac53]

so it looks like copyWith is seen as a crown jewel of the upcoming facility

The main reason we use this as an example is its well understood by many people, and it is also actually particularly demanding in terms of features to actually implement due to the public api itself needing to be generated :).

The language needs some mechanism of dealing with default values, which has been a showstopper in dart from day one.

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

[rrousselGit]

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

freezed supports this, but only because it relies on factory constructors and interface to hide the internals of copyWith (that is in fact a copyWith({Object? name = _internalDefault}))

[jakemac53]

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

Right, this is what I was describing which null safety actually does fix at least partially. You can make the parameter non-nullable (with a default), and then null can no longer be passed at all. Wrapping functions are required to copy the default value, basically it forces you to explicitly handle this does cause some extra boilerplate but is safe.

For nullable parameters you still can't differentiate (at least in the function wrapping case, if they don't provide a default as well)

[idkq]

Metaprogramming is a broad topic. How to rationalize? We should start with what gives the best bang for buck (based on use cases).

Draft topics for meta programming 'output' code:

1. Methods
2. Classes (shell)
3. Class members
4. Types
5. Enums
6. Statements (?)
7. Mixins (?)
8. Generics (?)

Also on output code:

Be able to visualize in some way the code generated into your program, at development time (https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md#usability)

Would be great if this could work without saving the file, a IDE-like syntax (hidden) code running continuously if syntax is valid. I refuse to use build_runner's watch

[porfirioribeiro]

Metaprograming opens doors to many nice features For the data class thing, this is something i miss from Kotlin. When i used Java we used @Data / @Value from Lombok that was some sort of generator, i guess having something like this would be enough for the data class's

Other language that does a great job at implementing macros is Haxe you can use Haxe language to define macros

I guess there are many challenges to implement this.

[ykmnkmi]

can we extend classes with analyzer plugin? can we use external and patch like patches in sdk libraries for extending classes? plugins for CFE?

[escamoteur]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

[jodinathan]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

I agree with this. I like the idea of meta programming as long as it doesn't remove how readable and maintainable a Dart code is.

[idkq]

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

[leafpetersen]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about. I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

[jodinathan]

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

In my honest opinion: things must be obvious, not magical. Every time I have to read or develop in PHP, JS or C with preprocessors etc... I just hate it. Too many magical stuff that you just can't read or debug easily. Dart is the opposite of that without being boring as hell as Java. In fact, there was a time that some Dart packages used to implement the noSuchMethod to create magical methods. Gee, what a pain. Meta programming could be the next Dart transformers if it takes the glittering magical road.

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

^ this

[esDotDev]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity.

But there is nothing beautiful about writing data classes or running complicated and and slow code-generation tools.

I'm hoping this can lead to more simplicity not less. Vast mounds of code will be removed from our visible classes. StatefulWidget can maybe just go away? (compiler can run the split macro before it builds?). Things can be auto-disposed. Seems like this could hit a lot of pain points, not just data classes and serialization..

[safasofuoglu]

Since dart currently offers code generation for similar jobs-to-be-done, I'd suggest evaluating potential concerns with that consideration:

• Metaprogramming is not simple/obvious - can it be made at least as simple/obvious as codegen through tooling?
• Metaprogramming will be abused - Is there a reason to think it will be abused more than codegen? (potentially, if it provides better ergonomics)

On the other hand, besides being an upgrade from codegen for developers, metaprogramming could provide healthier means for language evolution beyond getting data classes done. Quoting Bryan Cantrill:

Another advantage of macros: they are so flexible and powerful that they allow for effective experimentation. For example, the propagation operator that I love so much actually started life as a try! macro; that this macro was being used ubiquitously (and successfully) allowed a language-based solution to be considered. Languages can be (and have been!) ruined by too much experimentation happening in the language rather than in how it’s used; through its rich macros, it seems that Rust can enable the core of the language to remain smaller — and to make sure that when it expands, it is for the right reasons and in the right way. http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/

PS @jakemac53 the observable link leads to a private google doc.

[insinfo]

this would be fantastic if it allowed, the longed-for serialization for JSON natively without the need for manual code generation or reflection in time of execution

Today practically all applications depend on serialization for JSON, a modern language like dart should already have a form of native serialization in the language, being obliged to use manual codegen or typing serialization manually is something very unpleasant

[felixblaschke]

My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes.

#ToStringMaker() // '#' indicates macro and will rewrite all code in next scope
class Person {
    String name;
    int age;
}

// [REWRITTEN CODE] => displayed readonly in IDE
// class Person {
//    String name;
//    int age;
//
//    toString() => 'Person(vorname:$name, age:$age)'
// }

class ToStringMaker extends Macro {

    // fields and constructor can optionally obtain parameters

    @override
    String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
        var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code

        writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');

        return writer.code; // substitute code for referenced scope
    }

}

[idkq]

My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes.

Interesting. I like the structure a lot, but not so much #ToStringMaker() (hashtag working as annotation). Syntax would be better if #ToStringMaker() could be incorporated into the class definition.

[jodinathan]

It is very interesting. I find decorators not solid to something that changes the code, so I would add the macro within the class definition. The Macro class could accept a generic abstract type to show users what methods, properties etc expect from the macro along with full documentation capabilities. The abstract type would also be used by the analyzer/compiler to validate the macro result.

class Person macros ToStringMaker() {
    String name;
    int age;
}

abstract class ToStringDefinition {
   // some method documentation
   String toString();
}

class ToStringMaker extends Macro<ToStringDefinition> {
    // fields and constructor can optionally obtain parameters

    @override
    String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
        var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code

        writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');

        return writer.code; // substitute code for referenced scope
    }
}

[venkatd]

@jakemac53

This looks like a very useful feature!

What the Dart team picked the top 2-3 highest impact applications, built first-class implementations of them, and started with meta-programming itself as an implementation detail for the Dart language? This way you would all be free to refine meta-programming capabilities without making any public API promises to start.

[jodinathan]

I am not an expert in metaprogramming so I am not sure if dictating the macro with an abstract class is good or not, however, it fulfills my current need for source generation. I guess we could have a broader type of macro along with a more strict type of macro.

From my experience with source generation it is very unpleasant to work with strings when referring to properties and methods of a class, so I've changed the generate method a bit and added a new feature to print the properties of a class: ##Class.property. In our Person class of the example, if we did print(##Person.name) it would basically print name. The difference here is that ##Person.name can have a "go to definition" option and also be renamed when the original name property is renamed.

class Person macros ToStringMaker() {
    String name;
    int age;
}

abstract class ToStringDefinition {
   // some method documentation
   String toString();
}

class ToStringMaker extends Macro<ToStringDefinition> {
    // fields and constructor can optionally obtain parameters

    @override
    // [writer] is an inferred instance of Writer<ToStringDefinition> who knows all about this macro.
    void generate(Writer writer, MacroContext context) { 
        // ##ToStringDefinition.toString should print the string "toString" and the compiler should
        // correctly infer and print the return type and arguments based on 
        // the generic type "ToStringDefinition"
        writer.print(##ToStringDefinition.toString, '''
          ${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})
        ''');

    }
}

[felixblaschke]

From my experience with source generation it is very unpleasant to work with strings when referring to properties and methods of a class

So it's basically your abstract class is some kind of QOL when writing the generate method. You introduced some reference operator, so when writing a macro, we can use building blocks, to avoid hassling with strings too much.

In my mind the generated code shouldn't be visible to the user. The user should just see his physical code with some highlighting (because it got projected/transformed). Right next to it, the user should see the generated code that is collapsible by the IDE.

If a user clicks "goto definition" a clever algorithm should check, if that structure (he clicked on) is already existing in the physical code and goes there. If not the user will led to the generated code. In the example above: class Person, field name, field age will goto physical code.

Here are some other scenarios that came into my mind:

// Composing macros: executing from inside to outside
#Compareable() // adds toString(), hash(), etc.
#JsonSerializeable() // adds generic toJSON/fromJSON factory method
#AddFieldsFromJsonSchema(url: 'https://json-schema.org/learn/examples/address.schema.json') // applies class fields based on JSON schema file
class Address {}

// Macros need also work on all structures
#MeasureTime()
void someMethod() {
    // ...
}

#ProvideValue(url: 'http://math.com/constants/pi')
const int PI = 3;

Especially the generated Address needs to be incrementally compiled, cause it's field are referenced/used in other parts of business logic.

[jodinathan]

@felixblaschke In our company we have a source generator that scavenges models of the target package and create serialization and validation methods along with api CRUD and paging endpoints. With some arguments it also generates forms and dialogs, mostly for proof of concept. It is quite a big source generator.

From our experience meta programming could have:

• Templating: Working with string concatenation to generate code is not cool to read and maintain.
• Pin point: When you need to debug the macro for a specific part of the generated code, there could be a way to navigate to the exact part of the template that generated the code you need to analyze.
• Development only code: Macros could generate some or all code that is only used in development or production. Like a MeasureTime for dev performance purposes. It should also be tree-shakable.

All of your use cases are great and I would only think on how to add an inline macro scenario:

void foo() {
  #InlineMacro()
  bar();

  var daz = #OtherMacro() baz();
}

[felixblaschke]

@jodinathan I see your points. I agree that string concatenation offers not the best readability. But on the other side it's that what's happening. We are substituting code. In Flutter everything is Dart and there is no new other language, i.e. templating. But I understand that raw string generation is not the best. Maybe something like this:

// main.dart
#SomeMacro()
class Person {}

// Readonly generated code (refs are hidden in IDE ui)
class Person {            // #ref(some-macro.dart:8)
    String name = 'foo';  // #ref(some-macro.dart:9)
}                         // #ref(some-macro.dart:8)

// some-macro.dart
class SomeMacro extends Macro { // macros runs with runtime reflection to request own callstack to annotate line refs

    @override
    generate(MacroContext context) {
        var parser = DartCodeParser(context.source);
        if (parser.isClass) {
            var out = context.outputWriter;
            var myClass = out.addClass(DartClassParser(context.source).name); // everything that generates code can add a line ref
            myClass.addField('name', type: String, default: '\'foo\'');
        } else {
            context.fail('Macro only applicable on classes.')
        }
    }
}

The good thing about that is it's all Dart. Dart generates Dart. The helper tools (what's outputWriter in my example) can also be utilized by template parser. Theoretically you can write your own templating mechanism that falls back to this.

The syntax (as well as using #) is just for example proposes. I want to use something new with no existing meaning.

[idkq]

Users should be able to control/decide whether they want to see the real generated code or not. I personally would not for 90% of the time, but might need to. That should be configurable in the IDE. Maybe hover with the mouse to see the code (like we do to see class definitions).

We should be very careful of anything being inserted in someone's code since it can vandalize indentation, or anything else that the developer currently has full control of. This is highly personal.

[felixblaschke]

can vandalize indentation

Dartfmt is king :)

[Levi-Lesches]

I spent a while coming up with a design that I think will make a lot of people happy. I based it on above attempts (@felixblaschke @jodinathan) and the feature request doc (at the top of this issue) to see what the Dart team feels its priorities are. Some points that I found important from the doc:

Macros should:

• Macros should be written in normal imperative Dart Personally, I feel this is the same as not coming up with complex new syntax just for metaprogramming, since that's still twice the work and twice the maintenance. Use existing Dart functionality as much as possible
• Must be able to synthesize new APIs in existing classes.
• An easy way to invoke the metaprogramming
• Define some metaprogramming There needs to be a clean, type-safe way to declare what the new API will look like.

Macros should not:

• Modify existing code. I agree with this, and it is clear to imagine that changing code can get complicated.

Given these, I came up with the following for adding a .json getter to a class. Please give it a good read, feedback is of course welcome.

Say you have a file user.dart:

/// This is the data object we want to add `.json` to. 
///
/// Macros can use annotations for familiarity. So the `@ToJson` annotation applies
/// the [ToJson] macro to the [User] class. If that's too confusing, we can use a 
/// hashtag instead, so `#ToJson`. 
@ToJson
class User {
  final String first;
  final String last;
}

Then you have a macro defined in to_json.dart:

/// The interface that a [ToJson] macro would need to implement. 
///
/// Note that a programmer can manually make [User] implement this.
abstract class ToJsonInterface {
  /// Will be a map of all the class's fields in it. 
  Map<String, dynamic> get json;
}

/// A macro that extends a class with [ToJsonInterface].
/// 
/// By extending [Macro], it gets access to [Macro.mirror] (see below), which
/// provides access to the class's fields and functions. It specifies that it
/// will implement [ToJsonInterface], so it only adds to the existing code and
/// doesn't change it. 
class ToJson extends Macro implements ToJsonInterface {
  // Some syntax sugar would work well here, but that's a minor detail.
  ToJson(ClassDefinition sourceCode) : super(sourceCode);

  /// Here's where we actually define the `.json` getter. 
  /// 
  /// Here, the variables are interpolated `like this`. The resulting function 
  /// will be literal dart code. 
  @override
  Map<String, dynamic> get json => {
    for (final String fieldName in mirror.fieldNames)
      "`fieldName`": `fieldName`,  // --> "first": first, "last": last,
  }
}

This can be how macros are implemented under the hood.

/// The base for defining a macro. 
/// 
/// Here, I used `dart:mirrors`, to show how well it works with existing Dart.
/// Of course, it can also use some other system. 
class Macro {
  /// See [ClassMirror](https://api.dart.dev/stable/2.12.0/dart-mirrors/ClassMirror-class.html).
  final ClassMirror mirror;

  // [ClassMirror] doesn't currently take in any arguments, so that would have to change. 
  Macro(ClassDefinition classDefinition) :
    mirror = ClassMirror(classDefinition); 

  /// A simple getter, to show how to use mirrors in this context. 
  Iterable<String> get fieldNames sync* {
    for (final MethodMirror fieldMirror in mirror.instanceMembers)
      if (fieldMirror.isGetter) 
        yield fieldMirror.simpleName;
  }
}

Now, to revisit the points above:

Macros should:

• Macros should be written in normal imperative Dart
  • As you can see, adding .json to User was as simple as making User implement ToJsonInterface. This is how it's done currently.
  • Macros are defined by extending Macro and implementing some other interface. You can also use with.
  • Macro is a regular class with a mirror field. Maybe some helper methods but nothing ground-breaking.
• Must be able to synthesize new APIs in existing classes.
  • By specifying an interface and overriding APIs (like we did with .json), a macro can generate new APIs easily, and in a way that's familiar to programmers.
  • By interpolating variables in backticks, the values of those variables (usually the name of the fields) will be injected directly into the generated code.
• An easy way to invoke the metaprogramming
  • I used annotations syntax here. The very nature of annotations suggests they are capable of metaprogramming, which is why they are a natural choice. #-syntax can be used too.
  • Macros can be composed by simply stacking them on top of each other, which annotations already allow.

Macros should not:

• Modify existing code.
  • Existing classes will implement new interfaces, but existing methods won't be touched.

Instead of defining an entirely new syntax, it seems sticking with what Dart gives us has many benefits. Any thoughts? @jakemac53 @lrhn @leafpetersen

[Levi-Lesches]

I'm assuming that by printAllLists, you mean "print every element of every list", not just a and d particularly.

@PrintsAllLists()
class A {
  List<int> a;
  int b;
  int c;
  List<String?> d;
}

abstract class ListPrinter {
  void printAllLists();
}

class PrintsAllLists extends Macro implements ListPrinter {
  PrintsAllLists(ClassDefinition sourceCode) : super(sourceCode);

  @override
  void printAllLists() {
    // Again, this function uses dart:mirrors so this functionality all exists already. 
    for (final MethodMirror field in mirror.instanceMembers) {
      // Only print getters, not functions
      if (!field.isGetter) continue;
      // Only print lists, not other types
      if (!field.returnType.isSubtypeOf(TypeMirror(List))) continue;
      // Loop through the list and print it out
      // Here, we use backticks to show that this should be Dart code (it's awkward, see below). 
      // This is what should run as the generated function, the above is what generates it. 
      for (final element in `field.simpleName`) {
        print(element);
      } 
    }
  }
}

That should do it 🙂. Of course, it's not perfect, but I'm not trying to be.

The one thing I need help straightening out is the blurred line between "using Dart to find fields" and "using backticks to create Dart code". Here are my thoughts when writing this:

• Maybe instead of overriding printAllLists, the macro should return a void Function()? Just a thought, I'd need to expand on it more first.
• Maybe printAllLists should have a List<String> fieldNames paramater, and uses backticks to print all those fields. That way, you can separate between "macros using mirrors" and "using backticks to create dart code".
• EDIT: Double-backtick notation? (Yes, adding more backticks usually creates more confusion). Surround the code you want to be generated in backticks, and surround the (String) variables you want to be injected with double-backticks. So, the printAllLists function would look like this now:

  @override
  void printAllLists() {
  // Again, this function uses dart:mirrors so this functionality all exists already. 
  for (final MethodMirror field in mirror.instanceMembers) {
    // Only print getters, not functions
    if (!field.isGetter) continue;
    // Only print lists, not other types
    if (!field.returnType.isSubtypeOf(TypeMirror(List))) continue;
    // Loop through the list and print it out
    // Here, the backticks show Dart that this whole block should be generated. 
    // ``field.simpleName`` is injected into the dart code. 
    `for (final element in ``field.simpleName``) {
      print(element);
    }` 
  }
  }

which generates:

void printAllLists() {
  for (final element in a) {
    print(element);
  }
  for (final element in d) {
    print(element);
  }
}

I really like this. It requires a deviation from "pure Dart", but it seems there's no way to avoid it without introducing more problems. It also fits right in with the language feature request doc:

We propose to consider designing a quotation syntax so that macros that generate Dart code can generate code in a readable fashion. This also means code generators only depend on language syntax for code generation and not an imperative api, which will better stand the test of time.

[esDotDev]

Would this support auto-dispose of some objects, like:

_State {
  @disposeChangeNotifier
  FocusNode focus = FocusNode();
  SomethingElse someThingElse = SomethingElse();

 @override 
  void dispose(){
    // Macro should add dispose() if it does not exist in this method body, or create the entire method body if none is provided.
   <!-- focus.dispose()   -->
    someThingElse.dispose();
    super.dispose();
  }
}

[Levi-Lesches]

And my above notation means the JSON macro would be rewritten (a little more verbosely) as:

@override
Map<String, dynamic> get json {
  `final Map<String, dynamic> result = {};`
  for (final String fieldName in mirror.fieldNames) {
    `result ["``fieldName``"] = ``fieldName``;` 
  }
  `return result;`
}

It's a little more verbose, but also more clear to see what code is being generated and what is not. For example, the above macro will generate the following:

Map<String, dynamic> get json {
  final Map<String, dynamic> result = {};
  result ["first"] = first; 
  result ["last"] = last; 
  return result;
}

[Levi-Lesches]

@esDotDev Yes it will! (I'm using the backtick notation from above)

@Disposer
class MyWidgetState extends State<MyWidget> {
  // ...
}

/// This class extends State so it can copy the annotations from [dispose], 
/// such as the very-important [mustCallSuper] annotation. 
abstract class DisposeInterface extends State {
  void dispose();  // if you don't define a body, it goes back to abstract. 
}

class Disposer extends Macro implements DisposeInterface {
  @override
  void dispose() {
    `super.dispose();`
    // any other logic you have goes here
  }
}

[Levi-Lesches]

So, in summary, you need 3 components:

• a class to run the macro on
• an interface to define a (type-safe) API for the macro
• the macro itself, which generates code for every method it overrides in the interface. This separation is important since if you write a helper method for the macro (like getAllListFields), it won't show up in the generated code since it wasn't overriden from the interface.

[esDotDev]

Neat! A cpl other use cases come to mind:

• We might want to deprecate StatefulWidget, and just have UnifiedWidget or something. Could this split one class into 2? Essentially running the same macro the editor runs "convert to stateful widget"?
• We might want to create new classes from method signatures, another popular one for devs I imagine.

General thoughts on it: One benefit of this "keep it dart" approach seems to be a quicker development cycle. Anything that can significantly reduce the time this takes to get into users hands should be given extra weight I think. Flutter developers have needed Data Classes and Serialization for yrs now.

I also think the potential use cases drop pretty quickly in their level of impact with diminishing returns in regards to complexity:benefit ratio. Once we have Serialization, Data Classes, Auto-Dispose, less verbose StatefulWidget, the wins start dropping off pretty quickly for Flutter at least. So I think some fairly narrow focus here might be a good idea.

Even in that list, the first 2 items are by far the most important.

[Levi-Lesches]

@esDotDev

I also think the potential use cases drop pretty quickly in their level of impact with diminishing returns in regards to complexity:benefit ratio.

Completely agree. I hope that by making macros as easy to use as possible, we can stave off that point, but it does seem inevitable.

@tatumizer

the signature of the method you generate may also depend on the properties of the class. Yeah I sorta intentionally didn't get into that. In this case I agree with your implementation -- instead of overriding a method (and having the signature fixed), we can instead have a T generate() function, where T can be provided somewhere. I'll give it some thought. I'm not sure how type-safety is handled in static metaprogramming, but this will definitely play into that.

I started going into meta-metaprogramming, where you have a macro to define another macro, which seems very interesting. I had a function that determines a type T to be the signature and makes a new macro of type T... but I realized you can't pass Types around in Dart 😢. So the following doesn't work:

Macro getMacro() {
  final Type returnType = getType();
  return BetterMacro<returnType>();  // doesn't work
}

Until this works, we're pretty much limited to using backticks to define signatures. Though like you said, not impossible. Perhaps two types of macros (for now, until we unify them). One that uses an interface, and one that has a generate function. Personally, I feel the type-safety issue is too big to ignore (for now) to support the latter.

EDIT I really like the $-syntax, it's reminiscent of string-interpolation, which is what I was going for. Also resolves a lot of syntax/grammar ambiguities.

[Levi-Lesches]

I wrote about this issue of not being able to use Type variables as type arguments in dart-lang/sdk#11923, but again, this is a massive type-safety issue.

[idkq]

What about something like this

/*[[[codegen
fnames = ['DoSomething', 'DoAnotherThing', 'DoLastThing'];
for (final fn in fnames):
    print("void ${fn}();")
]]]*/
void DoSomething();
void DoAnotherThing();
void DoLastThing();
//[[[end]]]

[Levi-Lesches]

@idkq A good metric would be to compare against the feature request doc. Here are a few I pulled out:

Macros should:

• [ ] Macros should be written in normal imperative Dart -- the syntax is Dart, but it's in a comment.
• [x] Must be able to synthesize new APIs in existing classes -- yes, by capturing print output
• [x] An easy way to invoke the metaprogramming -- yes, simply include the macro in the source code
• [ ] An easy way to add macros together -- Maybe by putting them on top of each other?

Macros should not:

• [x] Modify existing code

I say the Dart-syntax is good, but maybe find a way to incorporate it into actual Dart code that can live alongside the source, instead of a comment. Also include some other function/syntax instead of print, and it could work. These two are what got me my extends Macro approach.

[jakemac53]

Would this support auto-dispose of some objects, like:

Yes we want to support this functionality.

[felixblaschke]

I made up another concept of macros to discuss / for inspiration. It's making use of a macro class in combination with a reference function, to reduce the string concatination.

// main.dart
class Person using Stringer() {
    String name;
    int age;
}

// stringer-macro.dart
class Stringer extends ClassMacro {

    int someParam;

    String({this.someParam = 42}); // constructor based configuration of macros

    @override
    void generate(ClassMacroContext context) {
        context.createMethod(
            name: 'toString', // name of created method
            returnType: String,
            parameters: [ // matches parameter in body reference function
                context.mirror.className,
                context.mirror.fields,
                Parameter(type: int, value: someParam), // just for API demonstration
            ],
            body: toJsonBody, // function for reference
        );
    }
}

// used in createMethod as a reference function to avoid too much string concatination
String toJsonBody(String name, List<Field> fields, int answer) {
    return "$name(${fields.map((f) => f.name).join(", ")}) ==> $answer";
}

// Example reference function body for dynamic return types
T fromJSONBody<T>(String json) {
    // ...
}

Edit: More explaination

The generated function will look like this:

class Person using Stringer() {
    String name;
    int age;

    String toString() {
        return "Person($name, $age) ==> 42";
    }
}

The generator creates a new method, but instead of writing all code with Strings, it uses some kind of template reference function that can do the job. In the generator you can specify all parameters that get pushed into the reference function. First parameter is the class name, second a list of all fields, and third a custom value (which isn't really required by a typical toString method). The reference function can be used as actual method, that gets projected on the Person class. I don't know if this is possible to implement into dart. First feedback looks like it's unintuitive. I wanted to create a way to parametrize the macro.

[Levi-Lesches]

I'm having trouble following -- what would the generated function look like in that case?