Make ANTLRParseTreeToPSIConverter aware of Rule subtypes

[vitorpamplona]

I'd like to propose a second "rule" processor (ANTLRParseTreeToPSIConverter) by the Rule Context's class name instead of the rule index in Antlr's generated Parser class. The goal is to provide enough information to the Parser Definition's createElement, though ASTNodes.elementType, to replicate the Context classes in the PSI Tree

Antlr has a generator option that includes subclassing rules by name. The grammar below generates a single Rule integer for simpleLiteral but three Context classes in the parser: SimpleLiteralContext, SimpleStringLiteralContext, SimpleNumberLiteralContext.

simpleLiteral
    : STRING           #simpleStringLiteral
    | NUMBER           #simpleNumberLiteral
    ;

SimpleStringLiteralContext, SimpleNumberLiteralContext extend SimpleLiteralContext

public static class SimpleLiteralContext extends ParserRuleContext {
    public SimpleLiteralContext(ParserRuleContext parent, int invokingState) {
        super(parent, invokingState);
    }
    @Override public int getRuleIndex() { return RULE_simpleLiteral; }
    //...
}
public static class SimpleNumberLiteralContext extends SimpleLiteralContext {
    public TerminalNode NUMBER() { return getToken(cqlParser.NUMBER, 0); }
    public SimpleNumberLiteralContext(SimpleLiteralContext ctx) { copyFrom(ctx); }
    //...
}
public static class SimpleStringLiteralContext extends SimpleLiteralContext {
    public TerminalNode STRING() { return getToken(cqlParser.STRING, 0); }
    public SimpleStringLiteralContext(SimpleLiteralContext ctx) { copyFrom(ctx); }
    //...
}

The change would need a RuleMap class as below:

// All Rule names, including subrules (subclasses)
val RULE_NAME_INDEX = antlrParser::class.nestedClasses.map { it.simpleName }.toTypedArray()

// List of Indexed Rule Types (required by IntelliJ)
private val RULES: List<RuleIElementType> = RULE_NAME_INDEX.mapIndexed { index, s ->
    RuleIElementType(index, s, CqlLanguage)
}

// Map between Context names and Rule Types
val RULE_MAP: Map<String, RuleIElementType> = RULES.associateBy { it.debugName }

And then a new ANTLRParseTreeToPSIConverter listener that would be very similar to the current one but with maker.done by Class name and not rule index.

class AntlrContextToPSIConverter(language: Language?, parser: Parser?, builder: PsiBuilder?) :
    ANTLRParseTreeToPSIConverter(language, parser, builder) {
    override fun exitEveryRule(ctx: ParserRuleContext) {
        ...
        marker.done(CqlRuleTypes.RULE_MAP[ctx.javaClass.simpleName]!!)
    }
}

The Parser Definition then create PSINodes for each subclass

   //...
   override fun createElement(node: ASTNode): PsiElement {
        val elType = node.elementType
        if (elType is TokenIElementType) {
            return ANTLRPsiNode(node)
        }
        if (elType !is RuleIElementType) {
            return ANTLRPsiNode(node)
        }

        return when(CqlRuleTypes.RULE_NAMES[elType.ruleIndex]) {
            "SimpleLiteral" -> SimpleLiteralPSI(node)
            "SimpleStringLiteral" -> SimpleStringLiteralPSI(node)
            "SimpleNumberLiteral" -> SimpleNumberLiteralPSI(node)
            //..
        }
    }

We implemented this variation in this project: https://github.com/Path-Check/intellij-cql

As you can see, we had to go around some of the interface design (Map of Rules instead of List) for the adaptor's converters to make it work.

Apologies for blending Java and Kotlin in the same text. :)

[parrt]

hi. So does this improve the ability to auto complete or something? I'm missing the goal in terms of the user experience. thanks.

[vitorpamplona]

In our case, it was the only way to add semantical attributes (return type) in the PSI three, which drove the auto completion work.

[parrt]

So it sounds like you're providing some thing internally that other programmers could use rather than something to user would experience? Is this something for my generic antlr jetbrains plug-in library or is this for the actual antlr plugin? Thanks... My memory is extremely fuzzy on this project

[vitorpamplona]

It would be an alternative to this class: https://github.com/antlr/antlr4-intellij-adaptor/blob/master/src/main/java/org/antlr/intellij/adaptor/parser/ANTLRParseTreeToPSIConverter.java

So, anyone else with an Antlr grammar that uses these rule subtypes can easily use this adaptor to build their plugins.

[parrt]

Oh sorry. I misread the repository. Yes this is the appropriate spot. Unfortunately I can't remember enough about what's going on to make a decision here... @bjansen ?

[bjansen]

@vitorpamplona I'm having trouble understanding why you can't use a rule index here:

marker.done(CqlRuleTypes.RULE_MAP[ctx.javaClass.simpleName]!!)

But in the parser definition you can:

return when(CqlRuleTypes.RULE_NAMES[elType.ruleIndex]) {

In the second example, won't the rule index be the same for SimpleLiteral and its two subclasses?

Never mind, you're building IElementTypes from all the class names, not from Parser.ruleNames.

[bjansen]

@vitorpamplona here's my first attempt: https://github.com/antlr/antlr4-intellij-adaptor/pull/28

There's a new constructor in ANTLRParseTreeToPSIConverter which you could use to provide your own list of IElementTypes for rules:

new ANTLRParseTreeToPSIConverter(lang, myParser, builder, new ClassNameIElementTypeMapper(CqlRuleTypes.RULES));