Question about local-expand mention in README

[jgarvin]

The core language does not coincide with the input language. Having an independent core language will hopefully allow us to overcome the overhead associated with recursive uses of local-expand, as well as enabling a second, trusted type checking pass.

Could you elaborate on this? What is the typical performance issue with local-expand, and how does separating the core and input languages help?

[david-christiansen]

In Racket, local-expand takes a piece of syntax, macro-expands it as if it occurred in the context (including the local scope) in which local-expand was called. The difference between local-expand and just expanding the syntax is that local-expand then returns the resulting piece of syntax to the macro's code, and that code can do anything to it. For instance, it might validate it somehow, rejecting code that doesn't live up to some criterion, or it might rewrite it. The macro should eventually return some code, but there's no guarantee that the code is exactly what came out of the expander (nor should there be - that would make local-expand much less useful).

The problem is that the macro expander needs to re-traverse code returned from local-expand, because identifiers in it may need to be re-resolved. For instance, the code that resulted from local-expand might contain an identifier x, and the macro that called local-expand may have placed it under a (lambda (x) ...). Depending on the specific lexical information attached to x, it may or may not refer to the lambda's x. This means that recursive macros that invoke local-expand end up with a quadratic overhead. Racket has syntax-local-expand-expression for the cases where there's just a validation step, but it doesn't help when introducing new bindings.

The idea in Klister that never got implemented was to have a separate, well-scoped-and-well-typed-by-construction core language representation that comes out of local-expand. This representation could be injected into syntax objects, allowing it to be returned from a macro, and all the macro expander would need to do is check whether the current context was a weakening of the embedded core language's context, which in many useful situations would be a pointer comparison, and that the types could match, which is just unification. Then, macros could observe and destructure this opaque core language representation using an API reminiscent of Edinburgh LCF, with actions in the macro monad to manipulate it while preserving well-typedness.

[jgarvin]

Thanks that's exactly what I was looking for :D