Separation of typed and untyped trees

[liufengyun]

• [x] basic def macros work
• [x] monadless works

[liufengyun]

As expected, monadless poses a challenge here:

val one = Option(1)
val two = Option(2)

lift { 3 * 6 + f(4) + unlift(one) + 3 + unlift(two) + 5 }

The transform needs to replace unlift(one) and unlift(two) in the typed tree with untyped identifiers, which are available from outer scope as ValDefs.

It's impossible to type check the identifiers first alone without type checking the outer scope definitions. On the other hand, if they are not type checked, it's impossible to interpolate them in typed trees.

[liufengyun]

Thought: typed and untyped constructors and extractors don't have to be the same form.

1. For Block, untyped: Block(stats: List[Tree]), typed: Block(stats: List[Tree], expr: Tree)
2. For If, untyped: If(cond: Tree, ifTrue: Tree, ifFlase: Option[Tree]), typed: If(cond: Tree, ifTree: Tree, ifFalse: Tree).
3. For Return, untyped: Return(expr: Option[Tree]), typed: Return(expr: Tree).

[liufengyun]

Ambiguity with constructors:

object Select {
    def apply(qual: TermTree, name: String): TermTree
    def apply(qual: tpd.Tree, name: String)(implicit c: Cap): tpd.Tree
}

implicit def tpd2untpd(tree: tpd.Tree): Splice = TypedSplice(tree)

def poly(a: String, b: Int): Int = meta {
  q"$a.toInt + $b"
}

q"$a.toInt + $b" has two different possible encoding:

toolbox.Infix.apply(toolbox.tpd2untpd(toolbox.Select.apply(a, "toInt")(gestalt.cap)), "+", toolbox.tpd2untpd(b))

or

toolbox.Infix.apply(toolbox.Select.apply(toolbox.tpd2untpd(a)(gestalt.cap), "toInt")), "+", toolbox.tpd2untpd(b))

Dotty favours the first choice, which doesn't type check.

[liufengyun]

One potential way to fix the owner chain without meta-programmer explicitly dealing with it: change the owner automatically in constructors during typed tree composition.

Conceptually, when one nests a typed def tree Inner inside another typed def tree Outer, the outer def tree Outer can call ensureOwner(subTree, Outer.symbol) to ensure the immediate children def trees point to itself.

Conceptually it seems it works, but I don't know why Dotty doesn't do that. There may be some caveats that I'm unaware of.

Mix typed tree in untyped tree can still be a problem. In Dotty, the owner chain rewiring of typed splice is handled by typer, which makes this case easy (need to check again).

[liufengyun]

Now the handling of owner is more robust f37a320.

@xeno-by Do you have any comment? I'd like to get this in, and implement https://github.com/scalamacros/macrology201/blob/part1/macros/src/main/scala/Macros.scala .

[xeno-by]

@liufengyun As we discussed on Friday, I don't think that the benefit of supporting macros like async and monadless outweighs the added conceptual and functional complexity introduced in this PR (typed AST construction, the notion of the owner chain, symbol construction, tree fixup).

It seems to me that our first release of a new macro system for Scala, Dotty and Intellij should only provide support for typed/untyped separation and untyped AST construction (including the notion of typed splices which is essential for automatic owner chain fixup).

In the meanwhile, async-like macros can be rewritten as compiler plugins or introduced as new language features via the SIP process. It is unfortunate that we have to reduce the power of the new macro system in comparison with the old one, but we envisioned situations like that, and I don't think these situations should stop us as long as the overwhelming majority of existing macros can be ported without much hassle.

If the demand in async-like macros ends up being significant and/or further research discovers a much easier way to write such macros, I'm up for changing my mind. After all, I am too very fond of crazy stuff like https://twitter.com/travisbrown/status/355799859120451584.

[xeno-by]

That being said, I don't think that this reasoning should stop research from moving forward. LGTM and best of luck!

[liufengyun]

Thanks @xeno-by !