[Macros] On writing files to disk.

[modulovalue]

I believe the current proposal implies that macros cannot emit any output to the filesystem, is that correct?

I am using a custom macro system that I would like to replace with the proposed macro system. I rely on my current macro system to produce Receipts that allow me to introspect and navigate generated code in a non-textual representation.

Here is an example that was produced from a specification for ANSI escape codes: ast.gen.pdf. The macro that lead to this receipt produces 20kloc. Browsing that code in a text editor is absolutely not practical.

I would also like to produce source map style output to be consumed by other tools. I don't see how the current proposal can support this use-case. I think users are going to want to give annotations arguments that aren't Dart code (e.g. SQL). Not having the ability to implement custom navigation to/from the provided SQL would be extremely painful.

[jakemac53]

I believe the current proposal implies that macros cannot emit any output to the filesystem, is that correct?

Correct, and there is no plan to add this. It is a lot more complicated than it sounds, and while there are some potentially valid use cases I think they are typically better done by some process external to the compiler itself.

For instance, we don't want people to do image scaling in the compile steps of their libraries. That is a recipe for disaster imo.

The macro that lead to this receipt produces 20kloc. Browsing that code in a text editor is absolutely not practical.

Generating documentation, visualizations, etc is not something we are targeting to support as a part of this feature, and I don't really think its something that should be a part of the compilation process generally. That is better done by a separate code analysis tool.

You could still use the analyzer, separately from the compiler, to generate these pdfs.

I would also like to produce source map style output to be consumed by other tools. I don't see how the current proposal can support this use-case. I think users are going to want to give annotations arguments that aren't Dart code (e.g. SQL). Not having the ability to implement custom navigation to/from the provided SQL would be extremely painful.

We have had suggestions for this previously, and I do think there is a valid use case there. We have not had time to consider what this would look like or how we would want to enable it. You will not have enough information yourself as a macro author to write the full source map, so outputting files would not help you (you don't know what the resulting macro augmentation library looks like).

What we might allow you to do is attach a source location to Code instances, and then generate a source map for you. Something like that I think is on the table for sure.

[modulovalue]

I think they are typically better done by some process external to the compiler itself.

That is not true. It seems to me that this statement can even be proven to be invalid for most useful cases.

The purpose of a macro (or in general a compiler) is to take a source domain A and to transform it into a target domain D. A compiler has to throw away information during the transformation from A to D. Compilation is therefore a function not a relation. The output D will never contain enough information to restore how D was created from A. Especially if the output is just a Dart file.

You could still use the analyzer, separately from the compiler, to generate these pdfs.

No. Compilation is a function, not a relation.

For instance, we don't want people to do image scaling in the compile steps of their libraries. That is a recipe for disaster imo.

I agree. However, the graph that I have provided is build by graphviz. The only requirement for that is to be able to output a text file containing valid graphviz .dot markup. Not even a millisecond of overhead.

---

It is a lot more complicated than it sounds.

Please elaborate.

I think you'd like to maintain the 'hermetic build' property and that is the main argument against file output (as well as performance) is that correct?

[jakemac53]

That is not true. It seems to me that this statement can even be proven to be invalid for most useful cases.

There are obviously tradeoffs, as with any system. But a system which has to regenerate arbitrary code documentation for all of its dependencies just to compile said dependencies, is a flawed system. A compiler should not be doing that work. It has nothing to do with running the program or preparing it for execution.

As an example, should generating docs/visualizations be a part of the hot reload workflow? That is a hard no from me.

If we expose a way to write arbitrary files, people will want to integrate things into the compilation step that really should not be integrated there, and should instead be generated offline, and checked into their source repo or otherwise published somewhere.

I think you'd like to maintain the 'hermetic build' property and that is the main argument against file output (as well as performance) is that correct?

• Hermetic builds are absolutely one of the challenges. The standard use case involves at least 2 separate programs running macros on any given app, simultaneously (the analyzer and the compiler). They will step on each others toes and cause problems.
• What happens if people manually overwrite these files?
• Where do the files get written? In the source tree? In .dart_tool?
  • If to source, do people check them in?
  • If to .dart_tool, what value does this really provide?
  • What if I want to generate files inside one of my dependencies? (or more likely, a macro runs on that dependency and wants to output a file). The pub cache is immutable, and the same package could be built with different dependencies, simultaneously.
• What if two macros want to write to the same file?
• General security concerns

That is just a few off the top of my head :)

[modulovalue]

As an example, should generating docs/visualizations be a part of the hot reload workflow? That is a hard no from me.

Oh, yes this is a good reason to be concerned. I think I see now where our disagreement lies. I will formulate one of my use-cases for macros in a different way that I hope will make some parts of my argument clearer. (sometime tomorrow).

---

A compiler should not be doing that work

Which part of the workflow pipeline for Googles Closure compiler generates the source maps? (Source maps were invented by that team, I believe, so this might provide valuable insight that I haven't considered.)

Also, JS sourcemaps for the Dart compiler are generated by the Dart compiler. I don't see how any other tool could be able to infer enough information from a compiled output to generate source maps?

---

people will want to integrate things into the compilation step that really should not be integrated there

I think users of macros should be free to make bad decisions and learn from their mistakes. They should not expect this type of handholding from the language team.

Where do the files get written? In the source tree?

Yes, alongside the 'to be augmented library' or some subdirectory inside of the directory of that file. The initial limitations could be that the name of each generated non-Dart file should start with the filename of the file that contains the 'to be augmented library'.

 '- my_dart_file.dart
 '- my_dart_file.my_custom_name

 or

 '- my_dart_file_macro_output/my_custom_name

If you don't want that, don't use that feature.

If to source, do people check them in?

Yes. If you don't want that, don't use that feature.

What if two macros want to write to the same file?

Can you think of an example where that would be the case?

Perhaps macros should be uniquely named. And to disambiguate in that case, generated files should contain that unique identifier

 '- my_dart_file_macro_output/my_macro_name/my_custom_name

What happens if people manually overwrite these files? & The standard use case involves at least 2 separate programs running macros on any given app, simultaneously (the analyzer and the compiler). They will step on each others toes and cause problems.

I believe build_runner requires that the filenames of the files that build_runner can generate have to be specified '''statically'''. That way the macro processor could know what to delete and when to rebuild. With that, the hermetic build property could be preserved?

General security concerns

Are there any security concerns that would not already exist from allowing macros to generate code? Macros could inject malicious code, so it should be assumed that macros come from a trusted source?

(Please point out things I haven't considered!)

[modulovalue]

The reason why you'd like to maintain the hermetic build property is efficiency, right? If the hermetic build property holds, then each build step can be processed by a different 'worker/thread' and so the build process can be made parallel?

---

We both would like to think of macros as 'people doing work at a company'.

I view them as 'Contractors', where I would like to have the ability to inspect what the contractor did and should I not like it, reject its output or simply take what the contractor did and fix it myself. Lets call this the Contractor Macro Model (CMM). For this, it would be best to store generated files on disk.

I think you would like to view macros as Employees. You trust them fully and you have no doubts that they can do their work efficiently. Should there be any problems, the employee could always be given feedback to fix the issue at hand. Lets call this the Employee Macro Model (EMM). It's just important that it works, not how it works, the details are not relevant. Having its output on disk is not necessary.

I would like to present evidence for the following: If you would like to have macro expansion, and therefore builds be as fast as possible, then EMM has fundamental limits as to how fast it can be (even in the presence of infinite computational resources). CMM will always be faster!

The fundamental limits are given by Amdahl's law. The relevant implication here is the following: a parallel system can, at best, only be as fast as its slowest unit.

Or in other words, with EMM macros, lets say we have 100 macro expansion tasks and 100 threads. Even if 99 of them take 0,1s, but one takes 10 seconds, the whole build will have to take 10 seconds. And there's nothing that can be done about that other than fixing that slow macro. The slowest macro expansion task will be the bottleneck.

We can remove this limit by having CMM macros, where files are stored on disk, and committed to source control. That way macros add zero additional overhead to the build system.

Edit: there was an issue at google once, that I think is relevant here: CppCon 2019: Chandler Carruth “There Are No Zero-cost Abstractions” It seems to me that the CMM model that I've proposed above would not make much sense in the context of C++, but Dart could provide improved performance guarantees compared to C++, and protect users by adopting CMM instead of EMM.

Edit: to be clear, the statements in this comment are not about writing any file to disk, just about writing expanded macros to disk. I agree, that writing arbitrary files to disk is not necessary but it would be a good feature, However it could be retrofitted in the future.

[jakemac53]

Checking in to source control the outputs of macros ultimately is not Sound for Dart. Macros have available to them information about transitive dependencies, but the way our package ecosystem works those dependencies may look different for different people running on the same version of your package.

The build package allows both models, because sometimes a builder might either know that it won't do that, or at least be confident enough about it to gloss over the potential unsoundness. This is why for example the Angular compiler builds to cache though.

It is actually an even more difficult problem than that, because you might have two different packages that end up checking in code generated by different versions of the compiler, which are not compatible. Now you end up having to version control your macros very strictly etc.

There are advantages to checking code into source control, I agree. But it isn't a good solution for macros imo.

[jakemac53]

Regarding efficiency, we already have to re-compile all transitive dependencies any time you check out your project on a new device. The amount of work Macros are expected to do is on the same order as that. So it isn't expected to make the situation any worse than the way it already is.

This is fwiw largely why pub can't ship dependencies pre-compiled as well. If we were able to solve this problem (lets say with a large distributed cache holding the compilation results of each library keyed by the exact versions of all deps), we would automatically get the same caching for macros. The macros should be expanded already in the dill files. And in fact internally this is exactly what will happen (because we use bazel).

[munificent]

We can remove this limit by having CMM macros, where files are stored on disk, and committed to source control. That way macros add zero additional overhead to the build system.

This doesn't fundamentally solve Amdahl's law. This is just a cache, and caches require invalidation. When the cache is invalidated, you're right back to taking as long as the longest build step requires to run. The only difference with caching the output to user-committed visible files is that (I presume?) you require the user to know when to manually invalidate the cache themselves. I've worked on many build systems over the years that require users to know when certain outputs are dirty and it always leads to pain.

I think there is another pair of perspectives you could have on macros:

• Input-centric: Any transformation process that takes Dart code as an input and requires the code to be analyzed before the transformation can be run. It can produce whatever output files it wants.
• Output-centric: Any transformation process that produces Dart code which ultimately ends up compiled and part of the final program being run. It can read whatever input files it wants.

It sounds to me like you're looking at macros from an input-centric perspective: you have a transformation step that analyzes Dart code, so you want it to be a macro. But we have designed it from an output-centric perspective. Macros are a tool for automatically generating Dart code (generally based on other Dart code, but possibly from other files) which is then compiled into a program.

Automatically generating architectural diagrams is an excellent thing to do, but it's not the kind of use case that macros were designed for. It's not a general purpose data pipeline system. It's a programming language feature for metaprogramming Dart.

[Levi-Lesches]

@jakemac53, while I agree with your overall view on macros, is there anything that can be done for users who don't use a Dart plugin with their IDE who still wish to see the generated code? Perhaps, if macros are not allowed to write arbitrary files, the compiler can output its generated files to a place of its choosing? And to @munificent's point, that doesn't necessarily imply that the files are cached -- nor the need for cache invalidation -- rather just having the code in a form that devs can inspect to see what code was generated.

[munificent]

@jakemac53, while I agree with your overall view on macros, is there anything that can be done for users who don't use a Dart plugin with their IDE who still wish to see the generated code?

It's a little out of scope for the language team because we try not to be in the business of telling the tools teams how the tools should work (since they're way better at that than us) but, yes, we are in favor of making the macro output visible to users in whatever ways the tool teams think makes sense. This is the main reason we're defining macros in terms of augmentation libraries: it provides a well-specified textual syntax that can represent macro output.

I wouldn't be surprised if our compilers ended up support some command line options to be able to dump macro output to a location of the user's choice in the same way that -S tells gcc to output the assembly code as text.

that doesn't necessarily imply that the files are cached -- nor the need for cache invalidation -- rather just having the code in a form that devs can inspect to see what code was generated.

If those files are read back by the compiler instead of being generated from scratch again, which is what I interpreted @modulovalue's comment on Amdahl's Law to imply then, yes, they are a cache. If they're just an output only for user debugging then I agree they aren't a cache, but they also don't help with compiler performance.

[modulovalue]

Thank you Jake and Bob for the responses. I will have to think about them before I reply properly.

---

@munificent wrote:

It's a little out of scope for the language team because we try not to be in the business of telling the tools teams how the tools should work

If you make a decision, waterfall style, that fundamentally limits what those teams can do, what do you expect for them to say if they can't implement the best solution due to a decision made that they don't have a say in at all? They will either have to risk their reputation or provide a suboptimal solution. I'm sure you're not trying to make that happen, but that's what I'm saying could happen here.

---

Thank you @munificent for providing the example of a compiler flag as a possible solution to the issues that I've raised. However, I do not see how that could work for the GitHub team. Consider the following feature that makes use of Scope Graphs. The GitHub team calls them Stack Graphs and here is a post that describes them: Introducing stack graphs | The GitHub Blog

Github uses Stack Graphs to provide efficient code navigation on GitHub. They allow them to do that in an incremental way. Something that the current proposal, even with a compiler flag, I believe could not support.

Relevant quote:

In each commit that we receive, it’s very likely that only a small number of files have been modified. We must be able to rely on incremental processing and storage, reusing the results that we’ve already calculated and saved for the files that haven’t changed. [...] This approach lets us create stack graphs incrementally for each source file that we receive, while only having to analyze the source code content, and without having to invoke any language-specific tooling or build systems.

Is my observation correct?

[jakemac53]

Github uses Stack Graphs to provide efficient code navigation on GitHub. They allow them to do that in an incremental way. Something that the current proposal, even with a compiler flag, I believe could not support.

Any build to cache Builder in build_runner already doesn't support this, fwiw. You can consider all macros as a similar pattern to build to cache build_runner Builders.

Even if github allowed us to run a build step to generate this graph, we would still technically have to be re-ran whenever a macro dependency changes (or transitive dependency of a macro dependency). But we may not even be compatible with that dependency, or it may not be a released version, etc.

Proposed solution

While it isn't perfect, what I would suggest is yes a mode where the compiler (or analyzer) output macro generated files to a known location. Or possibly it is a new tool to just do that, either way.

Then we would suggest that open source Dart repos enable that flag, and check that directory into their repository. They can have a CI check that it is up to date, to ensure they keep it reasonably updated (we may even provide such a github action).

This is effectively the same pattern as is used by build_runner today.

[jakemac53]

Note that there would be other benefits to such a mode/tool, users outside of IDEs can see the generated code more easily and it is viewable in github, etc. Even if it isn't technically fully up to date, it should be close. And it can easily be regenerated and checked in again.

[munificent]

@munificent wrote:

It's a little out of scope for the language team because we try not to be in the business of telling the tools teams how the tools should work

If you make a decision, waterfall style, that fundamentally limits what those teams can do, what do you expect for them to say if they can't implement the best solution due to a decision made that they don't have a say in at all? They will either have to risk their reputation or provide a suboptimal solution. I'm sure you're not trying to make that happen, but that's what I'm saying could happen here.

We meet with the implementation and tools teams every week. It's understood that there is a hard requirement that this feature continues to allow the great developer-time analysis experience that users expect from Dart.

It's definitely true that running Dart code which generates other Dart code complicates the static analysis experience. It's something we're sensitive to, but it's best approach we've found that fits our constraints and requirements. Dart users are already doing this with external code generation tools so macros arguably don't make things any worse and can make much of the user experience better by being more directly integrated into the tools.

This feature does mean that Dart isn't as amenable to simple textual static analysis as it used to be. In return, users can create powerful compile-time abstractions that I believe will make frameworks richer and more expressive and make application code significantly simpler and clearer. As always, language design is trade-offs and so far we're pretty optimistic that the increased complexity of analyzing Dart code is worth it.

[modulovalue]

munificent wrote:

[...] simple textual static analysis as it used to be.

That is not the problem that I'm referring to. Thank you for pointing that out, I'll clarify:

The analyzer implements many important features in linear time for which O(log(n)) time solutions exist (this is not obvious).

Take for example the following problem: Dynamic Connectivity. Fully dynamic connectivity concerns itself with implementing union find (a data structure that I assume the people here are familiar with) in a way that is incremental and decremental (that is, we can not only add, but also remove nodes). We have to trade the excellent O(alpha(n)) guarantees of union find for O(log(n)) but that still beats linear time asymptotically. But in practice, to not make the constants too large so that those theoretical results can be practical, we need our atomic units of information to be as small and to provide as much sound information as possible. I hypothesize that Dart could scale to billions of lines of code while supporting realtime analysis updates across changes within that codebase. If our atomic unit of information is a workspace, not a library, and we have to invoke a turing complete build step after each change then we're screwed. This is a problem that C++ macros, Rust macros, JavaScripts eval(), the C preprocessor, Kotlins Symbol processing API, Javas javac plugins have. But Dart doesn't have this problem yet. It doesn't have to, if the static macro implementation is designed and crafted carefully.

That's one of the reasons why code search inside of Google can't provide realtime updates to e.g. xrefs for newly checked in code (that was the case in 2015 or so, I presume that's still true today), but it could if more people had awareness of the theoretical bounds and would design their systems accordingly. But it takes one bad decision and then you're screwed. That's why I'm so persistent with this issue. I hypothesize it is doable with Dart today and I have been working towards that for some time.

Please, that above is important. If I'm not able to explain it in a way that is understandable please point that out.

The attentive reader will point out that cycles are a big issue in practice as well. Yes, but there are other theoretical results for datastructures that can help solve issues that involve cycles in O(log(n)). (e.g. augmented link cut trees and augmented euler tour trees (the state of the art is not on Wikipedia)). (We can even balance (balance as in BSTs) ASTs and plug them into a database and apply the big history of performance optimizations from the domain of databases directly to static analysis and even maintain indexes over ASTs. I am very excited about this topic, but I've been told before that others are not, so I'll stop here.)

---

I think Jakes proposed solution could work under the assumption that it is an opt-out, not an opt-in feature i.e. enabled by default across all of Darts public ecosystem.

Maybe it would be best to reach out to the treesitter team at github, that I believe works on parsing and building resolution results there, to see if it works for them. (See https://github.com/dart-lang/language/issues/813 & https://github.com/dart-lang/sdk/issues/37418 maxbrunsfield is the main author of treesitter)

---

munificent wrote:

We meet with the implementation and tools teams every week.

I'm glad to hear that, but

munificent wrote:

It's understood that there is a hard requirement that this feature continues to allow the great developer-time analysis experience that users expect from Dart.

While NNBD was executed flawlessly, other features, such as static extension methods were not. See: https://github.com/dart-lang/sdk/issues/38894 I hope you see how that makes me err on the side of overcautious regarding macros.

---

Personal note: This is hard for me, thank you to both of you for all the patience so far.

[jakemac53]

The primary goal I have been operating under is to not regress the existing functionality of our tooling, or that of any ongoing, planned optimizations.

That means supporting well Dart builds within bazel, as well as the incremental compiler (primarily used for flutter), whole world compilers (dart2js), and the analyzer.

None of these systems employ or have any plan or goal I am aware of to employ a strategy such as you suggest. We don't want to restrict the language and its expressiveness and usability for theoreticals.

Lets assume that I agree with your hypothesis (tbh, I simply don't have the time to try and even analyze it fully to form an opinion either way). That still wouldn't change my constraints or goals for this feature today.

@leafpetersen can maybe chime in if he thinks such a goal is something that he thinks is reasonable, or something we should design language features around, that is well above my paygrade frankly lol.

[munificent]

I hypothesize that Dart could scale to billions of lines of code while supporting realtime analysis updates across changes within that codebase. If our atomic unit of information is a workspace, not a library, and we have to invoke a turing complete build step after each change then we're screwed. This is a problem that C++ macros, Rust macros, JavaScripts eval(), the C preprocessor, Kotlins Symbol processing API, Javas javac plugins have. But Dart doesn't have this problem yet. It doesn't have to, if the static macro implementation is designed and crafted carefully.

In general, yes, it's definitely a good thing if all of the algorithms in your entire static analysis process have low algorithmic complexity. O(n) where n is the program size is acceptable, even lower is better. I know @leafpetersen and others on the language team have worked hard to make sure that things like type inference have good complexity bounds. When @lrhn and others worked on const in Dart, my understanding is that they tried to ensure that the const evaluation time and the resulting data size were both linear in the size of the input program (though I may have that wrong).

We also try hard to avoid whole-program analyses as much as possible for the same reason.

So, yeah, it's a thing we think about, a lot. You're absolutely right that as programs grow large, any superlinear algorithm running during static analysis is likely to become a problem. We have users that work on Dart programs that are, I think, on the order of millions of lines, though I'm not sure how often they have it all loaded into a single workspace.

With metaprogramming, we spent a good amount of time debating whether the metaprogramming language should be Turing-complete or not. My pitch at the time, which I think is likely still the right one, is that it should. The fundamental issue is the kind of use cases we're trying to support. Unlike other languages with macros, the goal for Dart is not to just let users define simple syntactic sugar where some input AST can be textually matched and expanded to something larger. It's not about simple stuff like lazy evaluation or implementing assert() in user code.

The use cases we have are things like, "generate an entire JSON deserialization implementation for a type and all of its fields, recursively." The metaprogram to implement that needs to introspect over the type its applied too, iterate over the fields, generate code conditionally based on the types of each field, recurse into their types, etc.

We could try to design a separate macro language that is not Dart, but is sufficiently expressive to handle use cases that complex and more. But my suspicion is that it would be very hard to design one as powerful as we need while still avoiding Turing-completeness. If you look at the kind of code Dart users are writing now in code generators, it would just be really hard to express that same behavior in a declarative non-Turing-complete language.

Based on that, we accepted that what we probably need is an imperative Turing-complete language for authoring macros. And, since Dart is an imperative Turing-complete language, it's the natural answer what language to use. It means we don't have to design, implement, and test an entire other language. Users don't have to learn an entirely new macro language. Users can reuse existing Dart libraries in their macros.

The trade-off is that, yes, it means static analysis requires running arbitrary user code, so there is no guaranteed bound on the time it will take. We believe that the trade-off is worth it, though it's certainly not without risk.

Note that macros only have access to the sound program and likely some additional set of known resource files. They can't access the file system freely, do networking, etc. That implies that a Dart implementation could execute all of the macros and cache the results of that to disk. Further static analysis after that should be as fast as Dart is today as long as the input program doesn't change.

[leafpetersen]

@modulovalue Thanks for the detailed discussion, it's very useful to hear from other perspectives here. I'm not 100% sure that I've internalized all of your concerns, so let me try to start by summarizing what I take to be the main point under discussion, which I believe is essentially described here (sorry if I'm missing other open issues, I think this thread has diverged a bit from the original opening, so I may have missed some things along the way).

I believe that your main concern as I hear it here is the desire to be able scale incremental analysis up arbitrarily, for some notion of analysis that at least includes understanding the static symbol references of a program. Your concern, if I hear you correctly, is that even understanding the symbols of a Dart program, with this change, will require running arbitrary Dart code to generate parts of the program. There are at least two problems I think you are pointing to here: first that running arbitrary Dart code can take arbitrarily long, and second that macros are non-local (that is, a macro might in principle walk arbitrarily far up the import graph and hence requires fundamentally non-local information to complete). Does that capture the main source of concerns?

I'm not entirely sure how to think about the implications of this, because I'm not sure how to bound your use cases.

For example, if we are considering analyzing a mono-repo (e.g. the google internal code base), then there is in fact a well-defined notion of what "the code" is. I'm not sure that macros deeply change the analysis problem in this case. At least at a pragmatic level, it seems reasonable to me to have the model that the results of running all of the macros are captured and reified as augmentation libraries, which then reduces the symbol resolution problem to being no harder (and no easier) than it is in the absence of macros. The only remaining issue is how to quickly and incrementally regenerate the cached augmentation libraries on any edits. This does indeed require a full Dart static analysis engine running on the edited code. Is that the main concern? You can't really do accurate symbol resolution anyway without running a full static analysis to infer the static types, which is also not particularly local, so I'm not sure this deeply changes the problem. In any case, I think an architecture here would have to be something like what I sketch: to incrementally update symbols, you can choose to solve the problem for the language without macros, and then separately you must solve the problem of incrementally running the macros to generate the cached artifacts whenever an edit happens. This latter doesn't seem too hard to me though? You already need to handle edits incrementally, the main difference is that now edits incur a second round of edits (that is, when you edit code, some macros re-run, which in turn generates some additional "edits").

If we're not in a mono-repo case (e.g. github), then I have to be honest, I don't really know how to think of analysis here at all. A standalone github repo is not closed: it refers to symbols which come from other pub packages, and pub is a level of indirection. Which actual code a symbol refers to is a function of how pub resolves a package URI, and there may be multiple valid and yet completely different choices for how to do this (completely different as in not even pointing to the same repository, I believe). So I'm not really sure what approximations are being made here to try to even get a start here? Though as I understand it, the github team is doing something.

I definitely feel that this is all somewhat ad hoc - it is already not the case that Dart is a language which is amenable to true separate compilation/analysis and this definitely makes it less so. But it does feel to me to be tractable enough in practice. Pragmatically, I'm not sure that scaling Dart analysis to billions of lines of code is very high on my list of things that our users really need, compared with things like better json serialization, data classes, etc; which they are very keenly asking for, and which we hope to provide via macros. So while I'm sympathetic to the concerns, I'm still of the opinion that we are making the right tradeoffs here.

Sorry if this misses the mark of your concerns - I'm trying to think through the various issues on the fly, and am not sure I have all of the right context loaded yet. Feel free to follow up with clarifications, or with concrete examples of issues that you think we should be addressing better.

[modulovalue]

@leafpetersen I think you have parsed my concerns correctly, thank you, except that perhaps our definitions of 'incremental' may differ which could be important so I'd like to clarify briefly:

I believe, your uses of 'incremental' refer to a system that is required to only be efficient with respect to new facts added to the system. i.e. the type of efficiency that union find provides. The validity of my arguments depends on a fully dynamic system i.e. a system that can be incremental and decremental i.e. a system that is guaranteed to be efficient not just with respect to new information added to it, but also with respect to updates to old information. Compilers throw all the intermediary structures that they've built up away, but If databases had to rebuild all their indices on every transaction that removed a row, that would be catastrophic to their performance, so I think that decremental efficiency is relevant and important to consider because many problems, such as dynamic connectivity, which is relevant to static analysis, do have solutions that are efficient decrementally.

You can't really do accurate symbol resolution anyway without running a full static analysis to infer the static types, which is also not particularly local, so I'm not sure this deeply changes the problem

We can be stateful, store partial results, combine those partial results later and reuse previous partial results when some of the results have been updated. GitHubs stack graphs do it that way. The research that they are based on have also solved that problem for various type systems such as System F. I see that as additional evidence that sound name resolution can be made fully dynamic for Dart, be highly efficient, and guaranteed to terminate. The static type system of Dart has been carefully crafted to guarantee termination (according to Erik), the current proposal would throw termination guarantees away, for features that other languagse have as first-class features or are already available through build_runner.

---

It doesn't matter from which angle I look at static metaprogramming, I can only find negatives with respect to the future potential of Dart.

We get:

• more convenient third party product types
• more convenient third party sum types and
• more convenient third party json serialization

but the downside is:

• it introduces novel constructs that users have no intuition for
• it adds an additional source of context sensitivity to the Dart grammar (if augment will get the same treatment as await).
• the semantics of dart will depend on its own semantics.
• we lose termination guarantees of static analysis.
• the dartpad team will need to be more careful about DDoS attacks or disable macros.
• And really nothing new will become possible with Dart because the people who need macros can already use build_runner or build a custom solution.

I have made this comment in one of the related issues and I still think that it is valid:

if we look at the history of programming languages, we can observe the following notable events:

Going from Machine code to Assembly added relative addressing. Removing Gotos by going from Assembly to C made the notion of basic blocks useful and non trivial. Going from C to Java added memory safety. Each of those steps are notable partially because they restricted a programming language (without sacrificing performance too much) for the possibility of significantly improved static analysis

dart:mirrors is being deprecated because it makes static analysis less efficient. Sealed classes and modules are being proposed because they allow for better static analysis.

Am I missing something fundamental? Am I stuck on a local maximum? Is my reasoning flawed? Leaf, Bob, can you point out something that I am not considering? Is there something else to this other than that 'it would make some users of Dart today happy'? :(

[munificent]

We get:

• more convenient third party product types
• more convenient third party sum types and
• more convenient third party json serialization

Those are some known uses cases, yes, but we expect this to support an unbounded number of uses. This is a bit like saying that adding higher-order functions only gives you map() and fold(). It gives you those plus an infinite number of unexpected novel uses that users will invent.

We expect that macros will become useful for libraries and frameworks in many ways, including some we can't predict.

• it introduces novel constructs that users have no intuition for

Compile time metaprogramming in one form or another is common in a variety of languages:

• The C preprocessor
• C++ template metaprogramming
• Scheme and Lisp macros
• Rust macros and procedural macros
• Java annotation processors
• C# CodeDOM

Etc.

• it adds an additional source of context sensitivity to the Dart grammar (if augment will get the same treatment as await).

This is a fairly minor annoyance and just seems to be how most languages evolve over time.

• the semantics of dart will depend on its own semantics.
• we lose termination guarantees of static analysis.
• the dartpad team will need to be more careful about DDoS attacks or disable macros.

Yup. These are real trade-offs.

The static type system of Dart has been carefully crafted to guarantee termination (according to Erik), the current proposal would throw termination guarantees away, for features that other languagse have as first-class features or are already available through build_runner.

Another way to look at this is that many Dart users are already using code generation. Which means their programs already have the exact property you fear: they can't be statically analyzed based on the source files on disk since the program there is incomplete until the code generation steps have run.

Having macros integrated into the language at least offers a well-defined semantics for those kinds of programs that users are already writing.

dart:mirrors is being deprecated because it makes static analysis less efficient.

Mirrors is deprecated not because it made static analysis slower, but because it made it less effective. In particular, it meant that dead code elimination, which Dart heavily relies on to reduce code size, is much less accurate. Compilers end up having to include a bunch of code just because it might be accessed at runtime through mirrors.

Macros solves that problem because now the metaprogramming happens at compile time before the compiler does its dead code elimination. It may make static analysis take longer, but the result should be a smaller compiled artifact.

[modulovalue]

First of all, I'd like to thank the language team for allowing me to express my concerns.

My goal was to highlight non-obvious ways how a static macro system will, in my view, jeopardize the future potential of Dart. I have provided theoretical results and technical arguments that were meant to point out issues that engineering will not be able to solve after static macros have shipped. It seems to me that Leaf has understood what I was trying to say.

I would like to conclude all the technical discussions with a short objective summary in a separate issue and close the existing ones.

@leafpetersen can I proceed with that or do you see how something that I have said so far is obviously wrong, in which case I'd like to take some time to think about that before I summarize everything.

[leafpetersen]

I would like to conclude all the technical discussions with a short objective summary in a separate issue and close the existing ones.

@leafpetersen can I proceed with that or do you see how something that I have said so far is obviously wrong, in which case I'd like to take some time to think about that before I summarize everything.

Sure, that's fine by me. We appreciate the careful engagement. I think we're largely disagreeing at the level of how to prioritize between different tradeoffs, which is.... a hard problem.

[modulovalue]

Thank you to everybody. See https://github.com/dart-lang/language/issues/2240#issuecomment-1142638309 for a short summary, I'm going to close this issue now.