set up instructions symbols in passes

[jimmyhchan]

Proof of concept for passes

[smfoote]

What does this achieve, aside from changing how the code is organized?

[jimmyhchan]

The meat of this is now in 3 different passes. Plus, we are currently only modifying the symbolsMap. Next step would be to modify the instructions list.

For example, right now the instruction open_tornado_body is a rather involved and calls out to buildTdBodyCode and requires a bit of knowledge including tdMethodName. Unfortunately, there is no place to put this data/logic. It ends up in Instruction util. Instead, what i'd like is for these compiler passes to expand the open_tornado_body into create_tornado_body_x, set_tornado_body_params, set_tornado_body_contents, end_tornado_body_x.

Another example, we currently keep track of the current state inside the instruction which means all compiler authors will be burdened with knowledge of all possible states. While we have many states, as a compiler plugin author do i really care about all state changes. What if I want to modify the meaning of a state change? Do i now need to change the core to get that done?. Instead, I'd like each pass to figure out what state means for itself and put those in instructions. Imagine if the insert_HTML_ATTRIBUTE instruction can be added to, modified or removed by some pass.

Imagine the desire to support shadow DOM creation inside of a tornado comment. The implementation would require us to keep track of STATE.TORNADO_COMMENT and STATE.HTML* but if these were just instructions, the compiler author who's adding this support can just add the instruction open|close_shadow_dom_body and then write the corresponding compilation logic.

[smfoote]

To be honest, I think you're trying to take the extensibility of the compiler too far. The insert_HTML_ATTRIBUTE instruction should not be allowed to be removed, because Tornado is an HTML templating language (see the Tenets). Too much extensibility is not a good thing; constraints are a good thing.

[jimmyhchan]

If we want to throw out bad attributes e.g. on click, there's no way to add that without modifing the core.

[smfoote]

You can throw out bad attributes before the instruction stage. Seems to me like that would be a transform on the AST.

[prashn64]

Before I dive in here, what's the goal?

[jimmyhchan]

@prashn64, Good question. This pull request is probably out of date, but the goal is fairly consistent.

Goals: 1) The API for compiler authors should be clear and unambiguous. Clear in that every instruction should define what state/or internal data changes need to happen and unambiguous in that the author of open_TORNADO_BODY should not have to worry about STATE. They should be able to write one thing and be able to unit test it.

2) the Instructions API should be extensible. As we start adding helpers and bodies and other features, they all go into the runtime because there is no place for it in the compiler (the compiler instructions don't change when we add a feature). Someone adding one feature will need to know how all of GenerateJS works and would need to understand how to modify the states/internal data without breaking things else where.

Proposed solution: Instead of one file (generateJS) which tries to implement all instructions, handle/resolve all state changes and all business logic, we build a low level API for a statemachine, a low level API for storage within each state, some storage across states and allow a bunch of files to use these APIs to translate instructions into JS code.

[smfoote]

The more I think about this, the more I think it is necessary. The @debugger helper, for instance, can't be turned into a compiler helper without something like this.

@jimmyhchan is this ready to be merged?