Syntactic macros

This proposal is a little different, it does not only try to propose a feature, but an implementation plan for the current compiler as well, since the task at hand is monumental. For more background information, please see the Metaprogramming summary issue.

I'd like to propose a syntax-level macro system for Draco, similar to Nim or Rust, primarily manipulating the AST. My reasoning is that while a metaprogramming system with semantic info can be useful sometimes, it is way more frequent that purely syntactic introspection is adequate and desirable.

The way we define macros would be identical to how we define functions. These functions would only be special in one way, they have to return an Ast, which is a package we ship with our BCL. This package would contain the type-definitions and factories for a syntax-tree structure that can be used to introspect and build Asts. Example for a macro definition and invocation that would log its parameter twice:

func twice(msg: Ast): Ast = Block(
    Call(Path("System", "Console", "WriteLine"), msg),
    Call(Path("System", "Console", "WriteLine"), msg));

Calling this macro like this:

twice!("Hello!");

Would expand to the following code (on an AST level):

{
    System.Console.WriteLine("Hello!");
    System.Console.WriteLine("Hello!");
}

The metaprogramming package

The package that would contain the Ast definitions would be a package we ship with our BCL. We could name it something like draco.meta. The AST structure provided would be a simplified version of our syntax trees, excluding things like unnecessary punctuation characters, parenthesis tokens or trivia.

Definition and invocation

Making macro declarations identical to regular to functions has a few advantages:

They can be trivially packaged into libraries
They stay reusable
They enjoy the same feature-set as regular functions

Macro invocations can take 2 forms: call-form and attribute-form.

Call form invocation

Call-forms use the syntax macro_name!(arg1, arg2, ...). The reason we do need to differentiate the call-site is because calling a macro is significantly different from calling a regular function, as the results will affect semantic checks. It is also beneficial to differentiate macro calls anyway, to not hide complex or funky behavior entirely inside an unassuming syntax.

Attribute-form invocation

Macros could be specified as attributes with the regular attribute syntax (not proposed yet):

@memo!
func foo() { ... }

Which would be equivalent to memo!(func foo() { ... }). The primary purpose for this seemingly redundant syntax is to simplify applying decorators to functions and types.

Additional arguments can be specified for the attribute macro, in case it requires them:

@cache!(Dictionary, TimeSpan.FromHours(2))
func foo() { ... }

Which would be equivalent to cache!(func foo() { ... }, Dictionary, TimeSpan.FromHours(2)).

Quoting

Manually building up the AST for bigger snippets can become painful. There is a reason Roslyn quoter exists. Similarly to Nim or Rust's quote!, eventually we should specify a mechanism to turn a snippet of code into an AST, allow interpolation, spreading, ...

The twice example could be rewritten to something like:

func twice(msg: Ast): Ast = quote({
    System.Console.WriteLine(#msg);
    System.Console.WriteLine(#msg);
});

Implementation plan

The feature is quite big, but fortunately can be sectioned into reasonable portions we can build and verify one-by-one in the compiler. Every step assumes the specification for that step is at least in the last stages.

1. Add compile-time evaluation capabilities to the compiler

While it might become a long-term goal to provide general compile-time evaluation, we need it internally for macros to execute. We do this either by executing our IR directly or translating to CIL and executing that.

The evaluation has to be constrained, for example can't allow the mutation of global state. We need to define these constraints (be a bit more strict now, we can relax them later).

The feature does not have to be exposed yet, but if we propose it until implementation, we can even roll with a publicly exposed feature.

2. Add the draco.meta package to our BCL

We need to define our middle-ground AST types and we need to be able to ship this as a built-in package. The compiler needs to be able to interpret these structures and convert them to syntax trees. The conversion to the other way is needed as well, we need to be able to take our syntax tree types and make simplified AST types from it.

Some utilities might come handy so end-users can play with it, like pretty-printing it, hooking it up with the formatter, ...

3. Wire these two together in the macro system

This step should connect up call-form macro invocations with the two. We should be able to transform each one of the arguments into the draco.meta structures, then call the comptime-evaluator to get back the substituted AST, turn it back into syntax trees, then let the compiler move on with the results.

3.5 Add attribute-like invocation

This is mostly sugar, after the previous step this shouldn't cause any difficulty.

4. Provide quoting

To wrap up the feature, we need to provide the quoting mechanism. It looks simple on the surface, just a keyword and some kind of parenthesis after it, we need to design and implement "holes" in it to be able to substitute arguments or local variables into it. Essentially interpolation on AST level. We might also want to provide a mechanism for other common things like spreading a collection of ASTs.

Draco-lang / Language-suggestions