dart-lang / language

Design of the Dart language
Other
2.65k stars 202 forks source link

Code generation (metaprogramming) proposal v2 #1565

Open Levi-Lesches opened 3 years ago

Levi-Lesches commented 3 years ago

This is my second draft of a proposal for code generation ("code gen" from now on) through specialized classes called "macros". The issue where this is being discussed is #1482 and my first proposal is #1507.

The key idea for me in code generation is that generated code should function exactly as though it were written by hand. This means that code gen should be strictly an implementation detail -- invisible to others who import your code. This further strengthens the need for code gen to be simple and expressive, so that any code that could be written by hand can be replaced by code gen, and vice-versa.

Another critical point is that code generation should not modify user-written code in any way. For this reason, part and part of are used to separate the human-written code from the generated. Partial classes (#252) help facilitate this, and this proposal heavily relies on them.

Definition of a macro

A macro is a class that's responsible for generating code. You can make a macro by extending the built-in Macro superclass. There are two types of macros: those that augment classes, and those that generate top-level code from functions. The Macro class consists of the generate function and some helpful reflection getters. The base definition of a macro is as follows:

// to be defined in Dart. Say, "dart:code_gen";

/// A class to generate code during compilation.
abstract class Macro {
  /// Macros are applied as annotations, see below.
  const Macro();

  /// The name of the entity targeted by this macro.
  String get sourceName => "";

  /// Generates new code based on existing code. 
  String generate();
}

Here are class-based and function-based macros, with Variable representing basic reflection.

Variable ```dart /// A example of how reflection will be used. /// /// This class applies to both parameters and fields. class Variable { /// The name of the variable final String name; /// Any annotation applied to the variable. final Object? annotation; /// Represents a variable. const Variable(this.name, [this.annotation]); /// The type of the variable. Type get type => T; } ```
ClassMacro ```dart /// A macro that is applied to a class. /// /// Generated code goes in a partial class with the same name as the source. abstract class ClassMacro extends Macro { const ClassMacro(); /// Fields with their respective types. List get fields => []; /// The names of the fields List get fieldNames => [ for (final Variable field in fields) field.name ]; // More helpers as relevant... } ```
FunctionMacro ```dart /// A macro that is applied to a function. /// /// Generates top-level code since functions can't be augmented. If you want /// to wrap a function, do it using regular Dart. abstract class FunctionMacro { const FunctionMacro(); /// The body of the function. /// /// => can be desugared into a "return" statement String get functionBody => ""; /// The positional parameters of the function. List get positionalParameters => []; /// The named parameters of the function. List get namedParameters => []; /// All parameters -- positional, optional, and named. List get allParameters => []; // More helpers as relevant... } ```

That being said, I'm not an expert on reflection so this is not really the focus of this proposal. I do know enough to know that `dart:mirrors` cannot be used, since it is exclusively for runtime. We want something more like [the `analyzer` package](https://pub.dev/packages/analyzer), which can statically analyze code without running it. Again, the point is that code gen should work exactly like a human would, and the analyzer can be thought of as a second pair of eyes (as opposed to `dart:mirrors` which is a different idea entirely). Depending on how reflection is implemented, we may need to restrict subclasses of `Macro`, since that's where all the initialization happens. ## Using macros To create a macro, simply extend `ClassMacro` or `FunctionMacro` and override the `generate` function. All macros will output code into a `.g.dart` file. The code generated by a `ClassMacro` will go in a partial class, whereas the code generated by a `FunctionMacro` will be top-level. Because the function outputs a String, `List.join()` can be a clean way to generate many lines of code while keeping each line separate. `dart format` will be run on the generated code, so macros don't need to worry about indentation/general cleanliness. To apply a macro, use it like an annotation on a given class/function. Here is a minimal example: ```dart // greeter.dart import "dart:code_gen"; // imports the macro interface class Greeter extends ClassMacro { const Greeter(); // a const constructor for use with annotations. /// Generates code to introduce itself. /// /// This function can use reflection to include relevant class fields. @override String generate() => 'String greet() => "This is a $sourceName class";'; } ``` ```dart // person.dart // Dart should auto-generate this line if not already present part "person.g.dart"; /// A simple dataclass. /// /// This is the source class for the generator. So, [Macro.sourceName] will be /// "Person". The class is marked as `partial` to signal to Dart that the /// generated code should directly modify this class, as though they were /// declared together. See https://github.com/dart-lang/language/issues/252 @Greeter() // applies the Greeter macro partial class Person { } void main() { print(Person().greet()); } ``` ```dart // person.g.dart part of "person.dart"; partial class Person { String greet() => "This is a Person class"; } ``` They key points are that writing the `generate()` method felt a lot like writing the actual code (no code-building API) and the generated code can be used as though it were written by hand. Also, the generated code is kept completely separate from the user-written `Person` class. ## Commonly requested examples: The best way to analyze any new feature proposal is to see how it can impact existing code. Here are a few commonly-mentioned use-cases for code generation that many have a strong opinion on.
JSON serialization ```dart // to_json.dart import "dart:code_gen"; /// Generates a [fromJson] constructor and a [json] getter. class JsonHelper extends ClassMacro { const JsonHelper(); /// Adds a Foo.fromJson constructor. String fromJson() => [ "$sourceName.fromJson(Map json) : ", [ for (final String field in fieldNames) '$field = json ["$field"]', ].join(",\n"), ";" // marks the constructor as having no body ].join("\n"); /// Adds a Map getter String toJson() => [ "Map get json => {", for (final Variable field in fields) '"${field.name}": ${field.name},', "};", ].join("\n"); @override String generate() => [fromJson(), toJson()].join("\n"); } ``` ```dart // person.dart import "to_json.dart"; part "person.g.dart"; // <-- added automatically the first time code-gen is run // Input class: @JsonHelper() partial class Person { final String name; final int age; const Person({required this.name, required this.age}); } // Test: void main() { final Person person = Person(name: "Alice", age: 42); print(person.json); // {name: Alice, age: 42} } ``` ```dart // person.g.dart part of "person.dart"; partial class Person { Person.fromJson(Map json) : name = json ["name"], age = json ["age"]; Map get json => { "name": name, "age": age, }; } ```
Dataclass methods ```dart import "dart:code_gen"; /// Generates [==], [hashCode], [toString()], and [copyWith]. class Dataclass extends ClassMacro { const Dataclass(); /// The famous copyWith method String copyWith() => [ "$sourceName copyWith({", for (final Variable field in fields) "${field.type}? ${field.name},", "}) => $sourceName(", for (final String field in fieldNames) "$field: $field ?? this.$field,", ");" ].join("\n"); /// Overrides the == operator to check if each field is equal String equals() => [ "@override", "bool operator ==(Object other) => other is $sourceName", for (final String field in fieldNames) "&& $field == other.$field", ";" ].join("\n"); /// Implements a hash code based on [toString()]. /// /// You can use more complex logic, but this is my simple version. It also /// shows that standard functions can be generated with macros. /// /// Make sure this is interpreted as [Macro.hash], not [Object.hashCode]. String hash() => "@override\n" "int get hashCode => toString().hashCode;"; /// Implements [toString()] by printing each field and the class name. /// /// Don't name it toString() String string() => [ "@override", 'String toString() => "$sourceName("', for (final String field in fieldNames) '"$field = \$$field, "', '")";' ].join("\n"); @override String generate() => [ equals(), hash(), string(), copyWith(), ].join("\n"); } ``` ```dart // person.dart import "dataclass.dart"; part "person.g.dart"; @Dataclass() partial class Person { final String name; final int age; const Person({required this.name, required this.age}); } void main() { Person alice = Person(name: "Alice", age: 42); print(alice.hashCode); print(alice.toString()); final Person alice2 = alice.copyWith(); final Person bob = alice.copyWith(name: "Bob"); if (alice == alice2 && alice != bob) { print("Equals operator works"); } } ``` ```dart // person.g.dart part of "person.dart"; partial class Person { @override bool operator ==(Object other) => other is Person && name == other.name && age == other.age; @override int get hashCode => toString().hashCode; @override String toString() => "Person(" "name = $name, " "age = $age, " ")"; Person copyWith({ String? name, int? age }) => Person( name: name ?? this.name, age: age ?? this.age, ); } ```
Auto-dispose ```dart // disposer.dart import "dart:code_gen.dart"; /// An annotation to mark that a class should be disposed. class ShouldDispose { const ShouldDispose(); } const shouldDispose = ShouldDispose(); /// Calls .dispose on all fields with the [ShouldDispose] annotation. class Disposer extends ClassMacro { const Disposer(); @override String generate() => [ "@override", "void dispose() {", for (final Variable field in fields) if (field.annotation is ShouldDispose) "${field.name}.dispose();", "super.dispose();", "}", ].join("\n"); } ``` ```dart // widget.dart import "disposer.dart"; part "widget.g.dart"; // <-- injected by code gen class MyWidget extends StatefulWidget { @override MyState createState() => MyState(); } @Disposer() partial class MyState extends State { @shouldDispose TextEditingController controller = TextEditingController(); // Not included in the dispose function int count = 0; @override Widget build(BuildContext context) => Scaffold(); } ``` ```dart // widget.g.dart part of "widget.dart"; partial class MyState { @override void dispose() { controller.dispose(); super.dispose(); } } ```
Functional Widgets ```dart // stateless_widget.dart import "dart:code_gen"; /// Creates a StatelessWidget based on a function and its parameters. /// /// This macro will generate the widget with the name [widgetName]. class WidgetCreator extends FunctionMacro { final String widgetName; const WidgetCreator({required this.widgetName}); @override String generate() => [ // because FunctionMacro generates top-level code, we can create a class "class $widgetName extends StatelessWidget {", // The fields: for (final Variable parameter in allParameters) "final ${parameter.type} ${parameter.name};", // The constructor: "const $widgetName({", for (final Variable parameter in allParameters) "required this.${parameter.name}", "});", // The build method: "@override", "Widget build(BuildContext context) {$functionBody}", "}" ].join("\n"); } ``` ```dart // widget.dart import "stateless_widget.dart"; part "widget.g.dart"; @WidgetCreator(widgetName: "MyButton") Widget buildButton() => ElevatedButton( child: Text(""), onPressed: () {}, ); void main() => runApp( MaterialApp( home: Scaffold( body: MyButton(title: "Fancy title") ) ) ); ``` ```dart // widget.g.dart part of "widget.dart"; // Notice how this is not a partial class, but rather a regular class class MyButton extends StatelessWidget { final String title; const MyButton({required this.title}); @override Widget build(BuildContext context) { // Notice how the => was desugared. return ElevatedButton( child: Text(title), onPressed: () {}, ); } } ```
## Implementation Since I'm not an expert on the Dart compiler, this proposal is targeted at the user-facing side of code generation. Anyway, I'm thinking that the compiler can parse user-code like normal. When it finds an annotation that extends `Macro`, it runs the macro's `generate` function and saves the output. Since more than one macro can be applied to any class/function (by stacking annotations), the compiler holds onto the output until it has generated all code for a given file. Then, it saves the generated code into a `.g.dart` file (creating partial classes when applicable), injects the `part` directive if needed, and compiles again. This process is repeated until all code is generated. Dart does not need to support incremental compilation for this to work: the compiler can simply quit and restart every time new code is generated, and eventually, the full code will be compiled. It may be slow, but only needs to happen when compiling a macro for the first time. Perhaps the compiler can hash or otherwise remember each macro so it can regenerate code when necessary. More detailed discussions in #1578 and #1483 discuss how incremental/modular compilation can be incorporated into Dart. This behavior should be shared by the analyzer so it can analyze the generated code. Thus, any generated code with errors (especially syntax errors) can be linted by the analyzer as soon as the macro is applied. Especially since generating strings as code is inherently unsound, this step is really important to catch type errors. Syntax highlighting can be implemented as well, but is not a must if the analyzer is quick to scan the generated code. A special comment flag (like `// highlight`) may be needed. How IDE's will implement "Go to definition" will entirely depend on how that is resolved for partial classes in general, but since these will be standard `.g.dart` files, I don't foresee any big issues. cc from previous conversations: @mnordine, @jakemac53, @lrhn, @eernstg, @leafpetersen ## FAQ - Why do you use partial classes instead of extensions? Extensions are nice. They properly convey the idea of augmenting a class you didn't write. However, they have fundamental limitations: 1. **Extensions cannot define constructors**. This means `Foo.fromJson()` is impossible 2. **Extensions cannot override members**. This means that `Disposer` and `Dataclass` wouldn't be possible. 3. **Extensions cannot define static members**. 4. **Extensions cannot add fields to classes**. I experimented with mixins that solve some of those problems, but you can't declare a mixin `on` a class that mixes in said mixin, because it creates a recursive inheritance. Also, I want generated code to be an implementation detail -- if we use mixins, other libraries can import it and use it. Partial classes perfectly solve this by compiling all declarations of `partial class Foo` as if they were a single `class Foo` declaration. - Why not use a keyword for macros? I toyed around with the idea of `macro MyMacro` (like `mixin MyMixin`) instead of `class MyMacro extends ClassMacro`. There are two big problems with this. The first is that it is not obvious what is expected of a macro. By extending a class, you can easily look up the class definition and check out the documentation for macros. The other problem is that if we distinguish between functions and classes, there's no easy way to say that with a `macro` keyword. By using regular classes, you can extend `FunctionMacro` and `ClassMacro` separately, and possibly more. This also means that regular users can write extensions on these macros if they want to build their own reflection helpers. Also, the idea of special behavior applying to an object and not a special syntax isn't new. The `async/await` keywords only apply to `Future`s, `await for` applies to `Stream`, `int` cannot be extended, etc. - Can I restrict a macro to only apply to certain types? This is something I thought about briefly. I don't see any big problems with this, and it could let the macro use fields knowing that they will exist. There are two reasons I didn't really put much work into this. Because I use macros as regular class, you can't simply use `on`. Would we use generics on `ClassMacro`? I'm impartial to it, but we'd have to have a lint to check for it since there is no connection between annotations and generics. Obviously this wouldn't apply to `FunctionMacro` or anything else. The second reason was that I want to encourage using reflection in code generation instead of simply relying on certain fields existing. For example, `Disposer` would be safer to write with this feature, but instead I opted to use reflection, and as such created an annotation that the user can apply to each field they want to dispose. And by using `@override` in the generated code, the analyzer will detect when the macro is applied to a class that doesn't inherit a `void dispose()`. - Why just classes and functions? What about parameters, fields, top-level constants and anywhere an annotation is allowed? I couldn't think of a good example. Reply if you have one and we can think about it. There were two reasons why it's unusual: One, reflection is a big part of macros. If there's nothing to reflect on, maybe code generation is not the right way to approach it. Two, macros should be logically tied to a location in the human-written code. Classes and functions were the most obvious. It's not so obvious (to me anyway) what sort of code would be generated from a parameter in a function. I suppose you may be able to extend `Macro` directly (depending on how reflection is implemented) and apply that to any code entity you like. - Aren't Strings error-prone? Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win! I'd love to hear feedback on this, but let's try to keep the conversation relevant :)
esDotDev commented 3 years ago

This is really cool, I appreciate the lack of magic, and the highly readable code.

It looks like it would be quite easy to support some special cases in auto-dispose as well, like:

// Usage
@shouldDispose(useCancel: true)
Timer timer = Timer();
... 
// Macro
String getDisposeMethodForType(Type type){
   if(type is Timer || type is StreamSubscription || useCancel) return "cancel";
   return "dispose";
} 
...
if (field.annotation is ShouldDispose){
    "${field.name}.${getMethodForType(field.type)}();",
}

This example could also be made more realworld if it supports null aware operators. It should write out controller?.dispose() if it's nullable, controller.dispose() otherwise.

Another thing to consider is how refactoring will work. If I re-factor MyButton via IDE, would the "MyButton" string be updated somehow?

It might be nicer for maintainability if a prefix-system were used here, like:

@WidgetCreator()
Widget createMyButton(){ // Some error could occur if create is not the prefix here
   ElevatedButton(
      child: Text(""),
      onPressed: () {},
    );
}

Where MyButton portion is a contract, that updates when either createMyButton or MyButton is renamed.

cedvdb commented 3 years ago

I like the syntax. Although:

Levi-Lesches commented 3 years ago

@esDotDev you can make it even simpler!

class ShouldDispose {
  final String methodName;
  const ShouldDispose({this.methodName = "dispose"});
}
// ...
@shouldDispose(methodName: "cancel")
Timer timer = Timer();
// ...
if (field.annotation is ShouldDispose){
    "${field.name}.${field.annotation.methodName}();",
}

If I re-factor MyButton via IDE, would the "MyButton" string be updated somehow?

Sorry I don't use an IDE with a "refactor" button, what exactly would it do? I'm trying to limit the magic here, so the name would be in the annotation and the body of the function is simply copied into the new widget. So if you refactor your widget code, the code gen will copy that new code into MyButton.

Widget createMyButton(){  // Some error could occur if create is not the prefix here

Again, to reduce magic, I don't want to introduce any new mechanism if we don't have to. An annotation would be enough, and you only declare (and maintain) the name of the widget in one place: in the annotation. Keep in mind, not all macros create widgets, so you can't introduce a new mechanism just for that.

Levi-Lesches commented 3 years ago

@cedvdb

I wouldn't use partial keyword as I wish we had Partial like typescript does for Record.

Refer to #252, not my idea

For the dispose example you might want to check if dispose exists on that annotated var, maybe not here, but i can see it being useful somewhere else

Well, it was just an example -- plus, I still don't have concrete details on reflection yet (I need to take a deeper look at the analyzer package). An important point of this proposal is that the analyzer should generate code, not just the compiler. That means that as soon as you mark a field as @shouldDispose, the corresponding field.dispose(); line will be generated, and an error will be shown if necessary. Since generated code is inherently unsound, this is the best way IMO to catch type errors. We can go a step further and say that all analyzer output for foo.g.dart should be shown in foo.dart, to allow devs to catch these errors even quicker.

Would it be possible not to have to write part "person.g.dart"; ?

Yes, I think Dart should automatically inject the part directive into person.dart (if not already present) as part of the code-gen step.

The .g file you mentioned are available only to the compiler and not actually created ? I don't use a specific runner library for that reason, it clutter the code base.

No, these files are actually created in the filesystem, totally accessible to the user. I mean, they'll probably go in the .gitignore, but they should be totally readable/inspectable to the user ("as if they had written it by hand"). When I tested my code, I had person.dart and person.g.dart in the same folder. For cleanliness, I'm not sure if that's what we want, but however we do it, it should be very clear (and at least in the part directive) where the generated file lives.

To be honnest this looks like a buider that is built in dart and runs before each compilation. I stand by my position that most use cases there could or even should be resolved another way, but if that means we get all those features all at once, so be it.

Yeah, that's pretty much exactly what this is. Although, hopefully we can optimize it to only run when we need it. I agree that most problems can be solved effectively with OOP and more functions, but there are some cases where the boilerplate just gets too much. But I'll bring up another point that @esDotDev once made: Making a StatelessWidget is not only boilerplate, it also clutters the codebase. It's much harder to read than a simple function, and it's only useful because of how Flutter is implemented. So you can have smaller functions that get expanded into full Widgets, but only the functions need to be maintained and read.

Same idea with toJson(). When you add a new field to a class, the compiler warns if you didn't add it to the fromJson constructor. But it doesn't tell you to add it to toJson(), and if you forget, you can corrupt your database. It's much safer to just maintain the fields themselves and let code-gen take care of the rest.

esDotDev commented 3 years ago

@esDotDev you can make it even simpler! Sorry I don't use an IDE with a "refactor" button, what exactly would it do? I'm trying to limit the magic here, so the name would be in the annotation and the body of the function is simply copied into the new widget. So if you refactor your widget code, the code gen will copy that new code into MyButton.

IDE lets you right-click on any Class or Method, and rename it, it scans the entire codebase and does a "safe" find and replace for those names. It's basically a glorified find and replace, with some sort of analyzer ability. http://screens.gskinner.com/shawn/d3ATmjLVlP.mp4

The importance as it relates here, is that it would make it very easy for anyone, anywhere in the codebase, to rename MyButton What happens to the "MyButton" string at that pt?

@esDotDev you can make it even simpler!

I think having the class handle super common cases like Timer and StreamSubscription manually is better than forcing developers to remember to type a magic "cancel" string, but we're into implementation details there, and all cases look easy enough regardless.

Levi-Lesches commented 3 years ago

Remember, WidgetCreator() and Disposer were just examples to show how you would make macros -- anyone can write a macro do whatever they want. You can use Macro.sourceName to get the name of the function and implement your own createMyWidget() pattern, or use reflection (once we figure that out) to make a safe AutoDisposer. The point is that it should be easy for everyone to do, and that's what this proposal tries to address.

esDotDev commented 3 years ago

Totally, no need to get into the weeds on implementation details, other than to reveal use cases.

In this case would be interesting to see how type checks would work (in general), and how null aware code might be written (in general). But I guess this is mostly a feature of the yet-to-be-written analyzer?

Levi-Lesches commented 3 years ago

Type safety is something I've pretty much given up on with code-gen. As you probably noticed from my first proposal, I tried very hard to shoehorn it in. It was a Macro class which generated any function declared in normal Dart and had a special @generate annotation. That way the signature can be preserved and the analyzer could keep it type-safe. But there were many problems with this:

  1. What if you don't know the name of the function in advance? (eg, custom getters/setters)
  2. What if you don't know the type in advance? (eg, custom getters/setters)
  3. What if you want to generate a constructor? (eg, Person.fromJson())
  4. What if you want to generate a top-level function/class? (eg, WidgetCreator)

All these, (particularly number 2!), showed me that there was no good way to handle this. That's why my new proposal focuses more on how to effectively integrate with the analyzer/regular way of writing Dart code. If the analyzer can instantly react when you apply a macro to a class, then that's essentially free type-safety. In other words, yes there will be a lot of type errors, but they'll be "errors" similar to the missing token } errors you get when you're still writing the code -- they're both easily fixable.

But I guess this is mostly a feature of the yet-to-be-written analyzer?

I was referring to the analyzer that currently comes with Dart. I plan to integrate this proposal specifically with the anlayzer package on pub.dev -- once I have time to figure it out. Based on what I hear around this repo, that package is hard to work with, so hopefully code-gen can motivate the right people to clean it up.

esDotDev commented 3 years ago

Sorry I just meant null aware code from the perspective of generation, like:

"controller${field.canBeNull? "?" : ""}.dispose();", // Outputs controller.dispose() or controller?.dispose()

Similarly with type, the macro might want to check for whatever reason, I'm assuming something like this would work? if(field.type is SomeType) doStuff

I really like the idea of instant code changes, and then just leaning on the compiler to flag errors. I'm still not sure who would be checking for errors on strings literals though... if MyButton class gets renamed to MyButton2 manually by a programmer (which will change all usages across the codebase to MyButton2), seems like the macro would just immediately pump out a new MyButton. Now your code is referencing MyButton2, which is de-coupled from the src method.

Levi-Lesches commented 3 years ago

Similarly with type, the macro might want to check for whatever reason, I'm assuming something like this would work? if(field.type is SomeType) doStuff

In my example, Variable.type is a standard Type variable. That's how dart:mirrors does it, I'm sure analyzer does it like that too. So yes, anything you can do today you can do with a macro.

if MyButton class gets renamed to MyButton2 manually by a programmer (which will change all usages across the codebase to MyButton2), seems like the macro would just immediately pump out a new MyButton. Now your code is referencing MyButton2, which is de-coupled from the src method.

Okay let's go a little more into detail for your example:

// function.dart
part "function.g.dart";

@WidgetCreator(widgetName: "MyButton1")
Widget buildButton() => ElevatedButton(/* ... */);
// main.dart
import "function.dart";

void main() => runApp(MyButton1());

Now, let's say we change MyButton1 to MyButton2. Depending on where we make the change, we get two outcomes:

  1. Change the parameter in function.dart: This means that function.g.dart now contains MyButton2, so all the references in main.dart are invalid. An error immediately pops up in main.dart. However, if you use "rename all" in your IDE (if you even can on a string), it should replace all instances, and there will be no error.

  2. Change the name in main.dart: WidgetCreator still contains MyButton1, so you'd get an error in main.dart saying there is no MyButton2. You'd realize that's because you didn't actually change the parameter in the generator. However, if you use "rename all" and your IDE replaces variable names in strings, it would also replace it in function.dart and you'd get no error.

All this can be summed up by saying that the code behaves to Dart as though it was written by a human. You as the developer still have to make sure you know what you're doing. If your IDE can help you, great. Otherwise, you just have to be a little careful. Or you can make WidgetCreator play nicely with your IDE.

As for the actual code gen, there are two "stages" of debugging generated code. The first is the macro writer, who writes the macro. The string literal itself, like you said, won't report any errors. But as soon as you put @MyMacro() on a class/function, it gets generated into Dart code, which the analyzer can inspect. Both the macro writer and the users who use it will be able to see these errors as if they had written the code by themselves.

esDotDev commented 3 years ago
  1. Change the name in main.dart: WidgetCreator still contains MyButton1, so you'd get an error in main.dart saying there is no MyButton2. You'd realize that's because you didn't actually change the parameter in the generator. However, if you use "rename all" and your IDE replaces variable names in strings, it would also replace it in function.dart and you'd get no error.

This is where refactoring with IDE comes into play. Rather than "rename in main.dart", you can think of it as "rename everywhere that class is referenced across the entire project". This means, when a dev hits F2, and Types MyButton2, the class in the .g file, it will also be renamed. The macro would then see that missing class, and presumably just re-generate it, just like it did the first time "MyButton" was declared.

At a high level, the IDE needs to tell the code gen system: This is what I just changed, and then the code-gen system could respond, but I don't think that is in scope of what you're proposing... or maybe it is? If you were to receive an event from IDE that "ClassA" is now "ClassB", could the declaration @WidgetCreator(widgetName: "ClassA") not be automatically re-written to use "ClassB"?

themisir commented 3 years ago

It would be great if we could somehow eliminate use of part 'something.g.dart' and .g.dart files. Instead of placing generated files on project the preprocessor could put those files on somewhere else like '.dart_tool' folder or in memory like other languages does. Otherwise I think meta-programming in dart will be same as source_gen-erators that runs on every build. I hope you get what I meant here.

Levi-Lesches commented 3 years ago

Otherwise I think meta-programming in dart will be same as source_gen-erators that runs on every build

Presumably, the compiler would be able to detect if the macro or the affected code changes, and only re-generate the code if it has. For example, if you import a macro that someone else made, it's very unlikely to change. That way, code-gen becomes a set-and-forget process, and won't impact the build process in the long-term. Where the generated code ends up going won't affect this process.

An often-requested feature of code-gen (which I agree with) is that generated code should not only behave like hand-written code, but should be just as accessible to devs, one reason being it makes debugging simpler. Which means not only inspecting generated code, but also seeing analyzer output, using an IDE's "Go to definition" feature, etc. In that sense it would be better to have a regular file. Dart is remarkable in it's lack of "magic", and hiding code from devs would really disrupt that.

Levi-Lesches commented 3 years ago

@esDotDev

This means, when a dev hits F2, and Types MyButton2, the class in the .g file, it will also be renamed.

Ah, here's where we were talking past each other -- you should never (even with current code-gen tools) edit a .g.dart file. You would change the annotation WidgetCreator(widgetName: "ClassA"), either by hand or with the "rename everywhere" feature. Then code-gen will automatically regenerate the widget, since it detected that the macro annotation changed. If I'm not mistaken, this is the choice that all code-gen tools have made -- can you edit a toJson() that was generated by json_serializable?

esDotDev commented 3 years ago

Ah I see. Looking at how functional_widget works, you can rename the Foo to Bar with IDE, and everything compiles fine. Once the code-generator runs again, it recreates the .g file, restoring Foo and deleting Bar which causes compile errors because the rest of the code is still referencing Bar.

This seems basically inline with what you'd expect, any changes to generated code are overwritten next time generator is run. A quality of life feature could potentially be built into the IDE to not allow refactor on classes that originate in a .g file.

Jonas-Sander commented 3 years ago

Aren't Strings error-prone?

Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive. Win/win/win!

  String generate() => 
    'String greet() => "This is a $sourceName class";';

Wouldn't it be more extensible to give back some kind of representation of the code which can be generated via a string or another way (codegen API)?

You still have all the benefits of the strings but still a way to use a "real" API later / as another way.

(don't know if AST is the right word)

AST generate(CodeGenerator generator) {
 return generator.fromString('String greet() => "This is a $sourceName class";');
}

then you could also later or as another API add more of a method based approach:

return generator.createMethod(
            name: 'toString', // name of created method
            returnType: String,
            parameters: [ // matches parameter in body reference function
                context.mirror.className,
                context.mirror.fields,
                Parameter(type: int, value: someParam), // just for API demonstration
            ],
            body: toJsonBody, // function for reference
        );
Jonas-Sander commented 3 years ago

@Levi-Lesches Is there any way you could fix the small typo in the title? 😁 ❤️

Code genertaion

Levi-Lesches commented 3 years ago

@Jonas-Sander:

Code genertaion

Sigh, no matter how much you review something mistakes always slip through! Fixed it, thanks.

Wouldn't it be more extensible to give back some kind of representation of the code which can be generated via a string or another way (codegen API)?

Well, two points on this: First, the way this proposal is set up, the Dart team doesn't have to build in any new code to teach the compiler what the AST is, or how to generate code methodically. All it has to do is write strings to a file, which is easily done with dart:io. In fact, I tested my example code by simply plugging it into DartPad and putting the generated code in a new file. All the functionality of error-checking and type safety is done by simply generating the code and letter the analyzer do its thing. So adding in an API would add a massive amount of work to code-gen.

Secondly, after discussing it both in #1482 and #1507, I think many people came to the conclusion that having an API can end up making code-gen messier, not cleaner. Let's try fleshing out a full example

@override
Code generate(CodeGenerator generator) => generator.createMethod(
  name: 'toJson',
  // Dart doesn't support nullables or generics as `Type` values. So you can't have Map<String, dynamic>
  returnType: Map,  
  parameters: [  // are these going to be used in the generated function?
    context.mirror.className,
    context.mirror.fields,
  ],
  // what goes here? 
  // Is it the actual function or a function that generates code based on `className` and `fields`?
  // If it's the actual function, how do you use variables like the name of the class and return type?
  // If it's a function to generate the code, you might as well use that instead of an API
  body: toJsonBody, 
);

VS

@override
String generate() => "";  // start writing code immediately. 

Some other issues with an API, as mentioned in the above issues:

  1. Such an API would have to keep up-to-date with new language features
  2. If types are not known in advance, there is no way to keep the API type-safe
  3. Understanding such code is harder than simply reading regular Dart code
  4. An API has to be expressive enough to cover every single use-case. It essentially has to duplicate Dart
  5. It has to be easy for all developers to use. Otherwise, we're back at square one.

In other words, I stand by my original comment:

Yes, but APIs are clunky and can quickly become out-of-date when new features/syntax are introduced. Strings reduce the learning curve, ease maintenance needed by the Dart team, as well as being maximally expressive.

If you have a specific use-case in mind that feels awkward with strings, please feel free to share.

Jonas-Sander commented 3 years ago

I completely understand your arguments "string vs AST".

I guess for me it just feels weird to have something as "low level" as a string directly as a return value from the generate method.

My argument is just that it would be more extensible to create the code via string-templating with something like my example of a CodeGenerator class. This API based approach was just an example to show that in this way you could still extend how you generate code without breaking old users.

Also what about some parameters to tune the code generation?

return generator.fromString(`...`, someTemplatingOption: true);

// Or
generator.setSomeOption(false);
generator.appendString(`...`);
generator.appendString(`...`);
return generator.code;

My point is that it AST was just some example of another Api that could thus added later in time without make breaking changes.

Is there a way to also set options by overriding getter in Macro class already? Yes.

May there be use cases where named arguments or methods on the CodeGenerator class make more sense? I don't know. Maybe you have some thoughts.

In the end my biggest concern was really just my gut feeling having something like a String given back. I know it makes sense if you only use string templating to write code. Still there may be some benefits to an approach as described.

renggli commented 3 years ago

I am surprised that all the discussions around Dart code generation remains at the level of string concatenation that is later parsed by the compiler. This seems unnecessarily low-level and detached from the actual language 👎🏻

Lisp and Scheme have a really nice concept around self-evaluating forms and quoting. I realize this is harder to adopt for a more complicated language like Dart; but I think it is worth to investigate and has successfully been done for other languages such as R, Rust, Scala, Smalltalk, Haskell, OCaml, C#, Julia, ...

Levi-Lesches commented 3 years ago

@Jonas-Sander, I'd agree with you if there was some someTemplatingOption that made sense, but I haven't been able to think of any. If you're talking about a parameter which changes the generated code, add a parameter to your macro's constructor. If you're talking about a parameter to change the formatting of the generated code, I would actually prefer that dart format be run automatically instead.

@renggli, based on that Wikipedia section you linked, it seems like "quoting" is just... string concatenation?

Both Common Lisp and Scheme also support the backquote operator (termed quasiquote in Scheme), entered with the ` character (grave accent). This is almost the same as the plain quote, except it allows expressions to be evaluated and their values interpolated into a quoted list with the comma , unquote and comma-at ,@ splice operators. If the variable snue has the value (bar baz) then `(foo ,snue) evaluates to (foo (bar baz)), while `(foo ,@snue) evaluates to (foo bar baz). The backquote is most often used in defining macro expansions.

I haven't used Lisp, so tell me if I'm wrong, but when you define x as (y z), Lisp is saving not only the values of y and z, but also their names. Later, when you use ` and `@, Lisp replaces x with the string y z, to be interpreted/evaluated by Lisp later.

I can understand to many that generating strings can feel wrong, but really everything is a string to begin with. When you write code in a .dart file, the compiler reads that as a string. No matter what form of generation you use -- API, strings, something like Lisp -- identifiers in your code are strings and have to be interpolated into your generated code somehow. Reflection is one way of understanding those strings at a higher level, but ultimately, generating strings as code is no different than typing by hand, which is what I was going for here.

Take the linked C# article:

Expression<Func<int, int>> doubleExpr = x => x * 2;

var substitutions = new Dictionary<ParameterExpression, Expression>();
substitutions[doubleExpr.Parameters[0]] = Expression.Add(
        Expression.Constant(3),
        Expression.Constant(4));

var rewritten = new ParameterReplacer(substitutions).Visit(doubleExpr.Body);
// (3 + 4) * 2

That's equivalent to:

String add(int a, int b) => "a + b";
String times2(String expression) => "($expression) * 2";
String generate() => double(add(3, 4));  // (3 + 4) * 2

The only difference is that there needs to be a way to convert strings back to code -- that's why this proposal uses .g.dart files.

There are also problems with using actual reflection (like tree-shaking) and type safety. While I'm not part of the Dart team, they've spoken up numerous times saying something like this would be incompatible with Dart today, or at least, very very hard. This proposal focuses on easing that strain to make code-gen more compatible with Dart and easier to implement, by treating code as strings, like humans would. Maybe this way, the Dart team may pick this up sooner rather than later. I'll quote @eernstg from #592:

We have discussed adding support for various kinds of static meta-programming to Dart, and it would be so cool, but it's a large feature and we don't have concrete proposals.

Now all that being said, I can certainly understand that this isn't going to satisfy a lot of people, and I encourage those who want to to try to find a way to fit code-gen more "natively" into Dart, but I would ask that it be in a new issue, because I consider it out-of-scope for this proposal. From Dart's perspective, they really are two entirely different problems -- generating strings and writing files is easily doable today, whereas full langauge support for metaprogramming would be a much greater effort.

Levi-Lesches commented 3 years ago

@tatumizer, based on your experience, do you feel this proposal would play nicely with the code you've written so far? Would it have made any parts of it easier? Harder?

esDotDev commented 3 years ago

This approach is really hard to argue with because:

  1. The important use cases in Flutter are small, and this addresses them all easily, and dart isn't really used outside Flutter.
  2. It requires very little work on the language side, so it will come to market very quickly
  3. Because it's string-based, there is very little maintenance to be done as the language grows and changes

So in the end, the argument for something more complex doesn't seem to justify it's existence. It would not open up any new use cases we can't already do, it would take much longer to get developed, it would likely have bugs for many mths or yrs of it's existence, etc etc all the fun stuff that comes with an order of magnitude more complexity. And for what? Compile safety on snippets that most developers never see, and are virtually never modified? It's not worth it.

Especially when you consider the test/debug flow of the proposed solution, it has instant string output, so as you are typing and saving, you could see the new code being generated, it would be extremely easy to debug, and the compiler would flag errors in realtime as you're hitting save.

Levi-Lesches commented 3 years ago

(e.g. if my class is declared as a template, the compiler should allow methods without bodies (this can be achieved by declaring them "external", but maybe it can be made the default for templates)

BTW, as defined in #252, partial classes won't allow you to have empty bodies. In a case where you don't have the body yet, you just don't include the header either, and define both later.

I can implement them with no framework in 100 lines of code or less based on mirrors and strings, and I can do it rather quickly

Yep, that's basically the point. This gets rid of the need for mirrors, since mirrors aren't really supported everywhere and have issues with runtime.

Trivial examples don't provide much material for generalization.

But then if you get too complex, you leave the field of code gen and end up back in Dart. Consider your example: insert a print("Starting function X"); and print("Finished function X") in every function whose name starts with "foo". Instead, you can do this:

T log<T>(T Function() func, String name) {
  print("Starting function $name");
  final T value = func();
  print("Finished function $name");
  return value;
}

String myFunc() => "Hello, World!";
String functionWithArguments(String name) => "Hello, $name";

void main() {
  final String message1 = log(myFunc, "Basic message");
  final String message2 = log(() => functionWithArguments("John Doe"));
}

With just a few lines to define log, we're able to wrap any function with it, with some benefits:

In fact, I would argue that only simple cases of just "I have to type the same thing again and again" should be handled by code-gen, and everything else with regular Dart.

esDotDev commented 3 years ago

A "typical" use case is just a gap in the language that many developers want plugged. They will be things currently served by code-gen right now. I believe these are pretty exhaustive: https://github.com/dart-lang/language/issues/1458#issuecomment-815155034, comes down to maybe 6 or 7 concrete use cases that Flutter devs would love to have today. (Maybe there are some on the pure dart side we're missing?)

I'm sure once we get used to Macros many other common use cases will emerge, but I really hope this doesn't get over-engineered. If it can be simple, let it be simple.

You wouldn't see code-gen doing the example described above because we can use a Mixin, Util Method or an Extension to add methods to things, and it would make no sense to generate code when good compiler enforced language constructs will do. Code-gen is about filling gaps, or reducing repetition, not replacing language features.

Levi-Lesches commented 3 years ago

Sorry, I meant that in some cases where someone would be tempted to use mirrors, like automatic toJson and fromJson, now they can use code-gen instead.

lrhn commented 3 years ago

@tatumizer That sounds like a job for aspect oriented programming, not code generation.

You want to figure out which functions starting with foo in package:something are called, and how often. That's a cross cutting concern, so you define a join point matcher targeting those (with some fancy pattern syntax) and put an "around" advice on each matching target, allowing you to wrap all invocations of those methods.

Worked pretty well for Java back when I used it some 15 years ago. Not exactly new technology 😁 .

Code generation is usually more about creating new classes (or other types) or methods, ones which cannot be simply inherited or mixed in by the existing language. It's parameterized or abstracted declarations, rather than instrumentation of existing code.

(Obviously a general "I can rewrite anything" framework can do anything to a program. It's a very big gun, and if you can do anything, doing something specific can get harder. A more specialized framework can help you with what it's specialized for.)

themisir commented 3 years ago

Not sure what you mean by that. Everywhere? Where do you need them to be supported? On Android cellphone? Why? For code generation, we need them to be supported only in one place: in the VM. Do you need them in javascript or smart watch?

Yes exactly, we'll only need mirrors during compile time to work with the code.

This might increase build time since the code will have to be parsed twice (before and after the code generation). But we might reduce build time by placing "code generation" part after code parsing (where expression trees are generated). Instead of generating code we could manipulate expression tree to include our generated code or even do some fancy stuff like enumerating all of the methods that name starts with "foo*" and add an expression that will log "foo method is called".

Also it will give another advantages since "expression tree modifying macros" will work faster (cause it doesn't have to be parsed multiple times) than regular string code generators, we might get advantage on editor support. The analyzer might execute macros and give intelligence to editors about changed code. Also it will eliminate creepy '.g.dart' files in project directory.

Would this will make it harder to introduce meta programming?

Wouldn't this make using meta programming harder?

What about maintainability?

themisir commented 3 years ago

Another idea, SDK might give us ability to modify built steps. So we can somehow execute our code between build steps (especially between tokenization and parsing or parsing and AOT/JIT) to make our own "meta programming" magic happen. This will not be modular as previous proposals but the communities might make their own meta programming patterns and later the the dart team might merge one of them to dart SDK, so Dart Team would have to do "less" work than designing modular meta programming interface.

This is all based on my assumption that Dart SDK is designed so it would support modularity.

lrhn commented 3 years ago

Yes exactly, we'll only need mirrors during compile time to work with the code.

What you want then is source mirrors. You don't need to reflect on run-time values, just source definitions. It'd be like an AST of the entire program, but without having to build the structure until you ask for it. It's also possible to have source mirrors which can't provide all the information (like not provide the source of function bodies in dependencies, only their signatures, which can then be supported by outlines for modular compilation).

lrhn commented 3 years ago

How do you draw a line between aspect-oriented programming and code generation?

Well, code generation generates code. It doesn't change existing code, it only generates more/new code. (And no, generating a /* before and a */ after existing code is not just code generation).

There are a number of phrases usually used in this domain.

Meta-programming is about having programs write programs. It can work using any of the approaches above, but the problems we're focused on there requires at least code generation. (Create built_value builders from the classes themselves. Create ==/hashCode/toJson/fromJson from fields.)

One big issue with code rewrite is that it's destructive. The moment you rewrite something, you need to re-analyze from scratch. Instrumentation usually doesn't change signatures. Code generation is usually additive, not transformative. That makes those approaches potentially more efficient (and obviously less powerful). The other big issue is that if the compiler rewrites the code, it's very hard for the author to know what happens by looking at the pre-rewritten source.

Levi-Lesches commented 3 years ago

What you want then is source mirrors.

That sounds like exactly what I'm looking for in this proposal. Is it accurate to say that package:analyzer is a source mirror?

Also it will eliminate creepy '.g.dart' files in project directory

What's so wrong with .g.dart files? You can inspect the code easily, just by opening it in your IDE. How would you debug a macro that writes straight to the AST? Taking from @lrhn's list, I would say the ideal is code gen and not the other two.

A big part of Dart is that there is no magic. What you see in the .dart file is what runs in your program, and the behavior is intuitive. Anything that modifies existing code will greatly complicate the language for minimal benefits. Imagine compiling a program but your code is completely overriden by macros -- but because it's directly in the AST, you wouldn't have any way to know. And once you do figure it out, there's no way to figure out exactly what was overriden and what replaced it.

With simple code gen, the scope of Dart programs doesn't change. It's not reflection that transforms your code into something you didn't write. It simply adds to your code in an entirely predictable way, and puts it in files that are completely transparent. It's no different than having a coding partner who knows exactly what you want, writes all their code in a second, and never falls behind with new changes. Nothing fundamental changed about your project, but it still massively reduces the code you have to write/debug if you can offload some to codegen.

ChristianKleineidam commented 3 years ago

When dealing with autogenerated code in the past I frequently had the problem of finding it hard to know what generated the code I'm looking at. Especially, when I'm working with existing projects where I don't know the structure.

If I as a programmer look at a g.dart file and I see a problem how do I go about fixing it? In the class Dataclass extends ClassMacro case it would be useful to be able to jump to the copyWith()-method of Dataclass from the copyWith()-method in the g.dart file.

Maybe it could look like:

partial class Person {

  @autogenerated("equals()")
  @override
  bool operator ==(Object other) => other is Person 
    && name == other.name 
    && age == other.age;

  @autogenerated("hash()")
  @override
  int get hashCode => toString().hashCode;

  @autogenerated("string()")
  @override
  String toString() => "Person("
    "name = $name, "
    "age = $age, "
  ")";

  @autogenerated("copyWith()")
  Person copyWith({
    String? name,
    int? age
  }) => Person(
    name: name ?? this.name,
    age: age ?? this.age,
  );
}

Then I could press Crtl+B in Android studio when having my cursor in equals() of @autogenerated("equals()") to jump to

String equals() => [
  "@override",
  "bool operator ==(Object other) => other is $sourceName",
    for (final String field in fieldNames)
      "&& $field == other.$field",
  ";"
].join("\n");

Pressing Crtl+B on autogenerated might make me jump to "@Dataclass()".

Such an @autogenerated("string()") annotation also has the advantage that it makes it very easy to see which of the functions in the g.dart file are generated and which are created by the user.

Levi-Lesches commented 3 years ago

I agree that some sort of jump-to would be useful. I was thinking to go from external code to generated code, but you're right that going from generated code to generator is useful too. Question, should jump-to take you to the annotation that invoked the macro or the macro itself? Keep in mind that in most cases users will import macros, so the generation code isn't actually theirs to control. Also keep in mind that some macros can have parameters, like WidgetCreator, that can change the generated code.

I think it would be more useful to jump to the annotation that applies the macro, especially since most macros will be imported. Also, I just remembered that the generated code will have a part of statement that shows which file caused it to be generated.

Here's another thing I haven't tackled: Where do these files go? By using part and part of, it doesn't matter, but every project will have its own needs. Do we allow full customization and default to something simple, or create our own convention? I'm in favor of taking inspiration from src and adding a generated folder in lib (that would be added to .gitignore) that mirrors the file structure of the lib folder. That way, users can find generated code really as if they wrote it themselves.

themisir commented 3 years ago

Just asking: is the new thing is rewrite of already exists "code generator"s? Or there's something new? Because if the new tool will let us to generate dart files using macros - isn't it just what build_runner is already doing? What's the point here? Maybe I'm missing a point or the tool will be just a simple string builder that uses analyzed data just like buildrunner. So we again have to write some magical stuff like `$SOMETHING`, because dart is "simple" - doesn't contains any magic instead contains less readable code like:

  Map<String, dynamic> toJson() => _$MyModelToJson(this);
  factory MyModel.fromJson(Map<String, dynamic> json) => _$MyModelFromJson(json);
themisir commented 3 years ago

Here's another thing I haven't tackled: Where do these files go? By using part and part of, it doesn't matter, but every project will have its own needs. Do we allow full customization and default to something simple, or create our own convention? I'm in favor of taking inspiration from src and adding a generated folder in lib (that would be added to .gitignore) that mirrors the file structure of the lib folder. That way, users can find generated code really as if they wrote it themselves.

We might store them in a temporary folder, maybe placed in project root (same level as pubspec.yaml located) or system level - idk. To prettify code, dart might introduce import shortcut to generated files so we can import them like:

import 'generated:models/mymodel.dart';
part 'generated:models/dependencies.dart';
ChristianKleineidam commented 3 years ago

As far as a place for the temporary folder, maybe the Build directory is a good place?

themisir commented 3 years ago

As far as a place for the temporary folder, maybe the Build directory is a good place?

I'm not sure if dart has a convention for "build" directory (dart != flutter). Isn't it related to flutter SDK?

Levi-Lesches commented 3 years ago

@TheMisir, that's why I want to wait for #252 to be implemented. Partial classes allow a class declaration to be split across multiple files. The JSON example is in the original comment of this issue, no magic at all.

I believe you're right that Dart doesn't have a build directory, and making one would be a pretty big conflict with Flutter and the like. I suggested inside the lib folder because the generated code is still Dart code, and the part/part of directives might get messy if the files are outside of lib. Also, any code that belongs to the user should be in a place the user expects it -- the lib folder.

EDIT I didn't notice your generated:dir/filename.dart, I like that better.

themisir commented 3 years ago

@TheMisir, that's why I want to wait for #252 to be implemented. Partial classes allow a class declaration to be split across multiple files. The JSON example is in the original comment of this issue, no magic at all.

I just get it. So, using partial classes we will be able to extend exists classes on generated files to add methods like toJson, fromJson, without having to modify original files.

And I liked it.

+1 from me

themisir commented 3 years ago

Also as I seen in C# partial classes also used for code generation stuff. For example when I design WinForms on UI designer the IDE generates code in Form1.Designer.cs which contains partial class of Form1 that creates and configures components as designed in visual designer.

Levi-Lesches commented 3 years ago

Yes, because we're working with the rule that we are not overwriting existing files, everything will go in a separate file with a part of directive. Functionally, it's the same as if it was written in the same file.

Levi-Lesches commented 3 years ago

We've gone back and forth on this a few times, but you're looking to generate data, I'm focusing on generating code. There has been talk about redoing the const mechanism to be more expansive, which, if done properly, can provide you with what you want.

In the case of generating code though, it's much safer to output it to a file where a human can inspect it easily. If that file isn't checked into version control, it's not burdensome at all and only provides benefits.

ChristianKleineidam commented 3 years ago

In the current proposal the only code that can be generated seems to be Dart code. For Flutter it would be useful to also be able to create code for other platforms.

If I for example could use @CrossPlatformDataClass to automatically create matching dataclasses with the same fields (and maybe a toJson/fromJson function) in Kotlin for Android and Swift for iOS that would cut down the required boilerplate code for cross-platform interop.

bwilkerson commented 3 years ago

If I for example could use @CrossPlatformDataClass to automatically create matching dataclasses with the same fields (and maybe a toJson/fromJson function) in Kotlin for Android and Swift for iOS that would cut down the required boilerplate code for cross-platform interop.

Independent of the question of whether to extend this proposal to include generating code in other languages, would it be useful to have a quick assist / code action to generate the boilerplate for the data classes? If so, please open an issue at https://github.com/dart-lang/sdk/issues/new to request one.

themisir commented 3 years ago

In the current proposal the only code that can be generated seems to be Dart code. For Flutter it would be useful to also be able to create code for other platforms.

If I for example could use @CrossPlatformDataClass to automatically create matching dataclasses with the same fields (and maybe a toJson/fromJson function) in Kotlin for Android and Swift for iOS that would cut down the required boilerplate code for cross-platform interop.

There's already work in progress project for doing that: https://pub.dev/packages/pigeon

ChristianKleineidam commented 3 years ago

As far as I know there are no existing quick assist actions that create new files and creating new files in a case like this has a few degrees of freedom about where to create them, that I would see as too complex for a quick assist action (but I will file issues for cases where I think quick assist would be helpful).

There's already work in progress project for doing that: https://pub.dev/packages/pigeon

When I read the existing introduction to metaprogramming essay, it articulates the desire for there being one metaprogramming system for dart. If I have to deal both with the pigeon system (which has no easy way to run during compilation) and with the dart internal system that adds unnecessary complexity. It would be great if the new system exposes a way for systems like pigeon to integrate with the main dart static metaprogramming and that means it needs API that let it generate files in other platforms.

bwilkerson commented 3 years ago

Independent of the same, can "quick assist" action be made smart enough to be able to not only generate, but also re-generate the said boilerplate on change? If so, maybe it can generate/regenerate other things (whatever they are), too? Maybe we have to talk about the mechanism of "quick assist/code action" plugins and count it as metaprogramming?

I don't believe that quick assists are a viable replacement for metaprogramming. There are plenty of limitations that make them inappropriate for that use. I only raised the question because we could potentially add a quick assist much sooner than we can ship a metaprogramming solution, and sometimes it's good to apply a bandage even when you still need to get stitches.

Quick assists can create new files (though I also can't think of any assists that do, only one quick fix). They could even update an existing file outside the one from which they are invoked. But they are fairly limited in some ways. Because of the way that most clients are written it's necessary for us to produce all of the changes at the time that we report that the assist is available. That means that performance is a concern (in order to be responsive), which in turn means that they generally need to be fairly local in scope. They also can't get any information from the user that isn't already in the source code. We could, for example, write an assist to generate methods to serialize and deserialize an object using JSON. We could not, on the other hand, write an assist to rename a class because (a) we couldn't ask for the new name of the class and (b) we couldn't, in a timely manner, search for all references to the class in order to update them.

bwilkerson commented 3 years ago

If something is missing, the assist can request more info by opening a dialog, or suggesting what should be added.

Actually, it can't. Being a language server we're limited to the functionality that the client/server protocol allows as well as any additional constraints imposed by the client. LSP has a mechanism (https://microsoft.github.io/language-server-protocol/specification#window_showMessageRequest) that allows a server to ask a multiple choice question, but nothing that allows for other kinds of information, such as an arbitrary string, to be returned. It's also the case that clients are not required to support this request so we can't depend on it. And our non-LSP clients also don't support anything equivalent.

If we had plugins then a user could write a plugin that defined the assist, disabled the assist when required information is missing, and generated a diagnostic to inform the user about the missing information. It could even generate a diagnostic when the generated code is out of date in order to inform users of the need to rerun the assist. But both of those depend on plugin support that isn't really ready for widespread use, and it wouldn't cover all the cases that a more general solution would cover.

Levi-Lesches commented 3 years ago

Also, fundamentally, quick assists are tools built-in to the IDE, which means they're a tool devs use for development. Code-gen, on the other hand, would be a part of the finished product and would stay in the code base. Meaning, WidgetCreator is not a shortcut to replace a function with a class, but rather it keeps the function a permanent part of the codebase, making it easier to maintain and understand. IDE tools would just generate the class and defeat the whole point.

They're two different tools with their own uses. Assists serve to, well, assist the developer in writing code, like a smarter auto-complete, whereas code-gen will bridge the dev's intent to the Dart compiler, similar to how OOP already does. I think we can all agree that even though import + TAB may be useful as a shortcut for importing flutter stuff, no one wants to see code that has import <TAB> and be expected to understand what that means. Both elements can be useful without replacing each other.