Serialisable TST/AST - an intermediate level before bytecode, JS

[sirinath]

I think there should be an intermediate distribution format in the form of a serialised AST representation which you distribute your packages as. Perhaps this format along with ByteCode and JS.

This representation has the following advantages:

• easy to create new backends
• the final byte code doesn't have to carry reified generics information - it can be represented at the TST/AST level
• helps in Meta Programming and language extensions
• multiple language versions will be compatible if the AST format says the same
  • easily reason about different versions of the AST format

Scala has realised the drawback in binary distribution and is moving towards such a representation called TASTY:

• https://docs.google.com/document/d/1h3KUMxsSSjyze05VecJGQ5H2yh7fNADtIf3chD3_wr0/edit
• https://docs.google.com/document/d/1Wp86JKpRxyWTqUU39H40ZdXOlacTNs20aTj7anZLQDw/edit

It might be good to consider this for Ceylon.

[gavinking]

I guess I don't really buy that this is very useful. We already have ceylon.ast, and the json-based model format in ceylon-js (that we should probably split out into its own package).

You've suggested:

• easy to create new backends

We already have a well-defined AST API that backends can use. I don't see how it's better to have a text-based format for this.

• the final byte code doesn't have to carry reified generics information - it can be represented at the TST/AST level

Well that's not really true. For Java interop we still need to store generics information in the Java bytecode. Anyway, we already have a separate json-based format for representing the "model", which is different to a full AST. A whole AST would be much too wasteful in this context.

• helps in Meta Programming and language extensions

Again, if you're thinking of stuff like macros, I think ceylon.ast already offers what is needed.

• multiple language versions will be compatible if the AST format says the same

Well that assumes that the grammar of the language changes more quickly than the grammar of the AST intermediate language. But I think that's exactly wrong. I think it's clear that in practice the grammar of the language, which is defined by the language specification, is much more stable than the grammar of this intermediate format.

• easily reason about different versions of the AST format

That's no easier than reasoning about different versions of the syntax of Ceylon, which is, as I say, defined by a spec.

[sirinath]

If you use a syntactic macro to build a DSL then need a common format to desugar in order to be imported and used within the language. Easier to work at a higher level like a serialisable AST IR than butecode. Then that compiler can change binary compatibility freely while the serialised AST format need not change.