Spec for Helper and other language extensions for compiler authors

[jimmyhchan]

This can be part of the constitution but a formal spec for compiler authors would also be helpful. IMO sections can also be thought of as extension of the core. I like the idea that everything is a helper or at least everything is an extension. By extension, I mean i can turn on/off a compiler flag and have it included/excluded from the output.

How should compiler authors think about helpers? Are there other extensions beyond helpers? How does one verify that the helper is compiled correctly? How does one verify that a compiler is correct?

[smfoote]

I like making everything an extension in some way, but I don't know about the idea of compiler flags and such. I think it would be great to make the language extensible enough that one can extend the compiler, but I think a lot of the extension would/should happen at runtime, which means the core runtime needs to allow for such extensions. I think that helpers are probably the most appropriate place for runtime extension.

I would like to remove the terminology of filters, because Tornado does not need "filters" as they are used in other templating languages. I think there still is a need for inline extensibility, but I don't think the name "filters" is the right one. Perhaps we can call them inline helpers, or something like that.

Finally, I think that verifying the correctness of compiled code is a separate issue and should be tracked separately.

[jimmyhchan]

the only reason that compiler correctness is relevant is w.r.t. figuring out how to implement helpers and other extensions. Agree that some things belong in the runtime, somethings belong in core and somethings belong elsewhere.

The issue for me is, in the old world, these things all kind of blurred (e.g. pragmas, removing whitespace with a runtime flag, helpers modifying the auto filters etc) an unfortunately, most things were shoved into runtime because the lifecycle for pushing it into the compiler was not there.

I can rename this issue to address specifically core compile time extensions: how does one build one and how does one check correctness. I'm pulling helpers into this because: I like the idea that there's only one API that even the built-in compile time code uses and, perhaps, it makes sense for some helpers to be compiled.

[jimmyhchan]

Influenced by a express middleware.

• tornado.preCompile.use(); // set up pre compiler (array of passes) where each pass accepts an AST -> AST
  • most precompile passes will use the AST visitor
• preCompile (aka buildInstructions); // this is not configurable.. if you want an instruction set it up in a pre-compile pass
• tornado.compiler.use(); // set up a compiler ( and instruction symbol map) where each pass accepts an InstructionList -> InstructionList
  • ... e.g. if we want to add pending into a section we would add that instruction here.
  • we would modify the codeGeneration walker to call and return InstructionLists.
• compile -> turns instruction list into executable based on the symol map
• tornado.helper.use(); // set up the helpers where each pass accepts a helpers object -> helpers Obj
  • this allows us to modify and/or extend helpers.