Silver concrete syntax construction syntax used in Silver extensions make post-refactor bootstrapping a nightmare

[krame505]

This is basically what we outlined in the COLA reflection paper, but the practical implications are much more annoying than I previously thought.

For example, @602p wanted to move the end keyword from silver:modification:let_fix to silver:definition:core to use this in host-language concrete syntax for origin tracking (where it probably belonged in the first place. This is a reserved keyword and is used by several extensions and modifications, some of which are importing the let modification just to use this keyword.)

However, the existing build of Silver uses Silver_Expr { ... } syntax literals in a number of places, many of which include let expressions and other syntax that uses the end keyword. So if one attempts to rebuild Silver normally after making this change, the silver:metatranslation library reflects the abstract syntax trees of these literals and then attempts to translate them into Silver code according to the current compiler's abstract syntax - however the new abstract syntax that is being compiled doesn't match that of the current compiler (the names have changed), so the build fails during the initial pass.

So the problem is that when bootstrapping this sort of refactor, we need to generate code for concrete syntax literals that matches the abstract syntax that we are currently compiling, rather than what we got from reflect. In the case of moving the end keyword, the process that I ended up following was

1. Apply this patch:

   --- a/grammars/silver/metatranslation/Translation.sv
   +++ b/grammars/silver/metatranslation/Translation.sv
   @@ -183,7 +183,7 @@ top::AST ::= terminalName::String lexeme::String location::Location
    terminalConstructor(
      'terminal', '(',
      nominalTypeExpr(
   -        makeQNameType(terminalName, top.givenLocation),
   +        makeQNameType(if terminalName == "silver:modification:let_fix:End_kwd" then "silver:definition:core:End_kwd" else terminalName, top.givenLocation),
        location=top.givenLocation),
      ',',
      stringConst(

2. Run ./self-compile --clean && cp build/silver.composed.Default.jar jars/
3. Make the change moving End_kwd to silver:definition:core
4. Run ./self-compile again to check that nothing broke
5. Revert the patch from 1
6. Finally run ./deep-rebuild
7. Commit and push the result, override the jars on Jenkins

We (I) will need to deal with this again, but on much worse scale when #152 finally gets done, moving compiler internals under silver: to silver:compiler:, since the full names of everything(!) will change at that point.

Note that this isn't a problem in the case of e.g. silver-ableC because there we have everything sorted out into a nice dependency graph, such that while we would like to just self-compile a change with the latest silver-ableC jars, we can always fall back to bootstrapping (abit slowly) from unextended Silver. One possible fix here would be to enforce the same sort of separation in Silver such that there would be 3 (or more) classes of Silver features/extensions:

1. Core silver, which can only use other core silver features (which may include modifications), and extensions that aren't permitted to use concrete syntax construction syntax.
2. 1 + the concrete syntax silver construction extension
3. 2 + all extensions, any literals used by these extensions can only reference features included in 1.

Then only 1 would need to be "permanently" bootstrapped (and could be rebuilt using the jars from 1, 2 or 3) while 2 and 3 can always be incrementally rebuild from the jars corresponding to just the features from 1. The obvious downsides of this would be

• Extensions that use features that they implement (e.g. strategy attributes) would have to be part of 1, meaning they can't use Silver concrete syntax literals.
• There would need to be at least 2 extra passes in deep-rebuild, so compile times would suffer. This could be mitigated by making deep-rebuild start using the jars from 3 and add a deeper-rebuild that does the full bootstrap which we only use for this sort of refactor or when things break.

So for these reasons I'm not in favor of making this change. I'm not sure there really is a nice solution here, but one option when we are performing big refactors like #152 is to follow the same procedure as above but instead of modifying silver:metatranslation, use the term rewriting library on the meta-translated Exprs to replace all names matching the old grammars with the new ones.

Anyway this issue (rant?) has gone on a lot longer than I intended, and this is mostly for me to reference in the future if this issue pops up again. So I'll stop here, but I am open if anyone else has thoughts on this problem.

[ericvanwyk]

Ugh. Thanks for ruining my Saturday evening... :)

So the proposal is to move end, but not do this larger restructuring you are describing at the end?

[krame505]

Yeah, we already did this as a part of the origin tracking patch (settled on the syntax attachNote note on e end.)

This restructuring is a mostly hypothetical exercise, which I don't recommend we actually do.

[ericvanwyk]

Ok sounds good. Thanks.