search

antlr / intellij-plugin-v4

An IntelliJ plugin for ANTLR v4
https://plugins.jetbrains.com/plugin/7358-antlr-v4
BSD 3-Clause "New" or "Revised" License
462 stars 103 forks source link

Control-click in ANTLR Preview gives `NoSuchElementException` #652

Open bseib opened 1 year ago

bseib commented 1 year ago

To reproduce this bug, quit IntelliJ and start up fresh. The bug is triggered only once, so the fresh startup is needed to see it the first time.

Using this grammar:

grammar Expr;
prog:   expr EOF ;
expr:   expr ('*'|'/') expr
    |   expr ('+'|'-') expr
    |   INT
    |   '(' expr ')'
    ;
NEWLINE : [ \r\n]+ -> skip;
INT     : [0-9]+ ;

Open the ANTLR Preview (i.e. right click prog in the source, and choose "Test rule prog")

Enter the following input:

1 + 3 * 3 - 5

Hold the CTRL key, and hover over the + character, and you should see the grammar rule tool tip appear saying #1 Type '+', Line 1:2:

image

While still holding the CTRL key in this spot, click the mouse button. An exception occurs with this stacktrace:

java.util.NoSuchElementException: token index 85 out of range 0..75
    at org.antlr.runtime.BufferedTokenStream.get(BufferedTokenStream.java:154)
    at org.antlr.intellij.plugin.preview.InputPanel.setCursorToGrammarElement(InputPanel.java:562)
    at org.antlr.intellij.plugin.preview.PreviewEditorMouseListener.mouseClicked(PreviewEditorMouseListener.java:46)
    at com.intellij.openapi.editor.impl.EditorImpl$MyMouseAdapter.runMouseClickedCommand(EditorImpl.java:3932)
    at com.intellij.openapi.editor.impl.EditorImpl$MyMouseAdapter.mouseReleased(EditorImpl.java:3850)
    at java.desktop/java.awt.Component.processMouseEvent(Component.java:6656)
    at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3385)
    at java.desktop/java.awt.Component.processEvent(Component.java:6421)
    at java.desktop/java.awt.Container.processEvent(Container.java:2266)
    at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5026)
    at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
    at java.desktop/java.awt.Component.dispatchEvent(Component.java:4854)
    at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4948)
    at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4575)
    at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4516)
    at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2310)
    at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2804)
    at java.desktop/java.awt.Component.dispatchEvent(Component.java:4854)
    at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:790)
    at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:739)
    at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:731)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
    at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
    at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:97)
    at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:763)
    at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:761)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
    at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
    at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:760)
    at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:667)
    at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.kt:615)
    at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:570)
    at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:68)
    at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:349)
    at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:348)
    at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:787)
    at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:348)
    at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:343)
    at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:995)
    at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:113)
    at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:995)
    at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$4(IdeEventQueue.kt:343)
    at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:831)
    at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:385)
    at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
    at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
    at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
    at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
    at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
    at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)

I'm using the latest IntelliJ:

IntelliJ IDEA 2023.1.3 (Ultimate Edition)
Build #IU-231.9161.38, built on June 20, 2023
Licensed to Broc Seib
Subscription is active until January 11, 2024.
Runtime version: 17.0.7+10-b829.16 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 13.4
GC: G1 Young Generation, G1 Old Generation
Memory: 2000M
Cores: 10
Metal Rendering is ON
Registry:
    debugger.new.tool.window.layout=true
    ide.experimental.ui=true
    eslint.additional.file.extensions=svelte
    ide.images.show.chessboard=true

Non-Bundled Plugins:
    org.antlr.intellij.plugin (1.20)
    org.rust.lang (0.4.197.5401-231)
    com.intellij.plugins.macoskeymap (231.7515.9)
    org.jetbrains.plugins.go (231.9161.14)
    com.intellij.nativeDebug (231.8770.15)
    String Manipulation (9.9.0)
    com.dherre3.wasm-text-parser (1.0.1)
    org.jetbrains.kotlin (231-1.9.0-release-358-IJ8770.65)
    com.github.bjansen.intellij.pebble (0.9.1)

Kotlin: 231-1.9.0-release-358-IJ8770.65
bjansen commented 11 months ago

@parrt I need help on this one. The code is basically this:

Token tokenUnderCursor = ParsingUtils.getTokenUnderCursor(previewState, offset);
Integer atnState = parser.inputTokenToStateMap.get(tokenUnderCursor);
Interval region = previewState.g.getStateToGrammarRegion(atnState);
CommonToken token = (CommonToken) previewState.g.tokenStream.get(region.a);

For some reason, atnState refers to a state that is not in the grammar, but apparently in a internal (generated?) grammar that looks like this:

expr
    :   ( {} INT<tokenIndex=41> 
        | '('<tokenIndex=45> expr<tokenIndex=47> ')'<tokenIndex=49> 
        )
        (
          {precpred(_ctx, 4)}?<p=4> ('*'<tokenIndex=20>|'/'<tokenIndex=22>) expr<tokenIndex=25,p=5>
                  | {precpred(_ctx, 3)}?<p=3> ('+'<tokenIndex=32>|'-'<tokenIndex=34>) expr<tokenIndex=37,p=4>
        )*
    ;

Then region refers to a location in this grammar instead of the grammar being tested, that's what causes the exception.

I have no idea how to fix this.

parrt commented 11 months ago

wow. that's a tricky one and I don't have a huge amount of time to dig into this. I wonder if the grammar and text window are somehow out of sync or grammar is delayed getting updated. I think I saw that at some point.

Ah. yeah, most likely it has something to do with conversion of left recursive grammars. Anytime there is a left recursive grammar we convert it to a new version of the rules which is where the atnState will be referring to. going backwards to the location in the original grammar would be tricky but on guessing I have a mechanism to do that mapping if only to produce error messages from the command line. I guess we will need a check in the plugin looking for left recursive rules and, if so, look in the original grammar for the position not inthe generated grammar. I would start by poking around and seeing how I generate error messages from the command line for left recursive rules.