low priority: comments intermittently fail to highlight syntactically

[ghost]

Hi, I apologize for a vague issue as I still haven't figured out how to reproduce, but if you start typing a lot of comments eventually the syntax highlighting breaks without recovery. This can also be triggered by pasting code into lalrpop files with syntax errors, seemingly randomly.

It may take me weeks till I even realize how to reproduce as it's honestly quite perplexing, but write enough comments // or paste enough code into a lalrpop file and this will happen to you.

I say this is low priority though since you can fix it simply by restarting intellij. So not a blocker, just an annoyance.

The following error logs are shown, though I'm not sure if or how they are related:

Caused by: java.lang.IndexOutOfBoundsException: Wrong offset: 2209. Should be in range: [0, 767] | at com.intellij.openapi.editor.ex.util.SegmentArray.offsetOutOfRange(SegmentArray.java:89) | at com.intellij.openapi.editor.ex.util.SegmentArray.findSegmentIndex(SegmentArray.java:103) | at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.incrementalUpdate(LexerEditorHighlighter.java:178) | at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.documentChanged(LexerEditorHighlighter.java:334) | ... 74 more | 2021-08-28 19:21:29,522 [ 257640] ERROR - enapi.editor.impl.DocumentImpl - IntelliJ IDEA 2021.2.1 Build #IC-212.5080.55 | 2021-08-28 19:21:29,523 [ 257641] ERROR - enapi.editor.impl.DocumentImpl - JDK: 11.0.11; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o. | 2021-08-28 19:21:29,523 [ 257641] ERROR - enapi.editor.impl.DocumentImpl - OS: Linux | 2021-08-28 19:21:29,523 [ 257641] ERROR - enapi.editor.impl.DocumentImpl - Last Action: SelectNextOccurrence | 2021-08-28 19:21:29,523 [ 257641] ERROR - enapi.editor.impl.DocumentImpl - Current Command: Typing | 2021-08-28 19:21:29,525 [ 257643] ERROR - enapi.editor.impl.DocumentImpl - com.intellij.openapi.editor.ex.util.LexerEditorHighlighter(com.mdrobnak.lalrpop.LpLexerAdaptor): Error updating after DocumentEventImpl[myOffset=2279, myOldLength=0, myNewLength=1]. | com.intellij.openapi.editor.ex.util.LexerEditorHighlighter$InvalidStateException: com.intellij.openapi.editor.ex.util.LexerEditorHighlighter(com.mdrobnak.lalrpop.LpLexerAdaptor): Error updating after DocumentEventImpl[myOffset=2279, myOldLength=0, myNewLength=1].

[ghost]

this offset error seems to also disable diff views (comparing local history). I can do unified view but not side-by-side.

Not sure if I should open a separate issue as I'm guessing its same root cause. Here's the error:

2021-08-28 22:40:07,711 [4906397] ERROR - diff.impl.DiffRequestProcessor - Unexpected termination offset for lexer FlexAdapter for com.mdrobnak.lalrpop.lexer.LalrpopLexer | java.lang.IllegalStateException: Unexpected termination offset for lexer FlexAdapter for com.mdrobnak.lalrpop.lexer.LalrpopLexer | at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.doSetText(LexerEditorHighlighter.java:461) | at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.setText(LexerEditorHighlighter.java:420) | at com.intellij.openapi.editor.impl.EditorImpl.setHighlighter(EditorImpl.java:1346) | at com.intellij.diff.util.DiffUtil.setEditorHighlighter(DiffUtil.java:196) | at com.intellij.diff.util.DiffUtil.configureEditor(DiffUtil.java:263) | at com.intellij.diff.tools.holders.TextEditorHolder.create(TextEditorHolder.java:97) | at com.intellij.diff.tools.holders.TextEditorHolder$TextEditorHolderFactory.create(TextEditorHolder.java:107) | at com.intellij.diff.tools.holders.TextEditorHolder$TextEditorHolderFactory.create(TextEditorHolder.java:101) | at com.intellij.diff.tools.util.side.TwosideDiffViewer.createEditorHolders(TwosideDiffViewer.java:106) | at com.intellij.diff.tools.util.side.TwosideTextDiffViewer.createEditorHolders(TwosideTextDiffViewer.java:115) | at com.intellij.diff.tools.util.side.TwosideDiffViewer.(TwosideDiffViewer.java:53) | at com.intellij.diff.tools.util.side.TwosideTextDiffViewer.(TwosideTextDiffViewer.java:67) | at com.intellij.diff.tools.simple.SimpleDiffViewer.(SimpleDiffViewer.java:63) | at com.intellij.diff.tools.simple.SimpleDiffTool.createComponent(SimpleDiffTool.java:31) | at com.intellij.diff.impl.DiffRequestProcessor.createState(DiffRequestProcessor.java:269) | at com.intellij.diff.impl.DiffRequestProcessor.lambda$doApplyRequest$4(DiffRequestProcessor.java:344) | at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:705) | at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:647) | at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:63) | at com.intellij.openapi.progress.impl.CoreProgressManager.executeNonCancelableSection(CoreProgressManager.java:223) | at com.intellij.diff.impl.DiffRequestProcessor.lambda$doApplyRequest$5(DiffRequestProcessor.java:338) | at com.intellij.diff.util.DiffUtil.runPreservingFocus(DiffUtil.java:757) | at com.intellij.diff.impl.DiffRequestProcessor.doApplyRequest(DiffRequestProcessor.java:328) | at com.intellij.diff.impl.DiffRequestProcessor.lambda$applyRequest$3(DiffRequestProcessor.java:310) | at com.intellij.openapi.wm.impl.FocusManagerImpl.lambda$doWhenFocusSettlesDown$3(FocusManagerImpl.java:184) | at com.intellij.util.ui.EdtInvocationManager.invokeLaterIfNeeded(EdtInvocationManager.java:101) | at com.intellij.ide.IdeEventQueue.ifFocusEventsInTheQueue(IdeEventQueue.java:186) | at com.intellij.ide.IdeEventQueue.executeWhenAllFocusEventsLeftTheQueue(IdeEventQueue.java:140) | at com.intellij.openapi.wm.impl.FocusManagerImpl.doWhenFocusSettlesDown(FocusManagerImpl.java:175) | at com.intellij.openapi.wm.impl.FocusManagerImpl.doWhenFocusSettlesDown(FocusManagerImpl.java:181) | at com.intellij.openapi.wm.impl.IdeFocusManagerImpl.doWhenFocusSettlesDown(IdeFocusManagerImpl.java:41) | at com.intellij.diff.impl.DiffRequestProcessor.applyRequest(DiffRequestProcessor.java:318) | at com.intellij.diff.impl.DiffRequestProcessor.applyRequest(DiffRequestProcessor.java:297) | at com.intellij.diff.impl.DiffRequestPanelImpl$MyDiffRequestProcessor.updateRequest(DiffRequestPanelImpl.java:107) | at com.intellij.diff.impl.DiffRequestProcessor.updateRequest(DiffRequestProcessor.java:180) | at com.intellij.diff.impl.DiffRequestProcessor$DiffToolToggleAction.actionPerformed(DiffRequestProcessor.java:683) | at com.intellij.openapi.actionSystem.ex.ActionUtil.lambda$performActionDumbAwareWithCallbacks$4(ActionUtil.java:240) | at com.intellij.openapi.actionSystem.ex.ActionUtil.performDumbAwareWithCallbacks(ActionUtil.java:261) | at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAwareWithCallbacks(ActionUtil.java:240) | at com.intellij.ui.popup.ActionPopupStep.performAction(ActionPopupStep.java:252) | at com.intellij.ui.popup.ActionPopupStep.performAction(ActionPopupStep.java:242) | at com.intellij.ui.popup.ActionPopupStep.lambda$onChosen$2(ActionPopupStep.java:228) | at com.intellij.openapi.application.TransactionGuardImpl.performUserActivity(TransactionGuardImpl.java:94) | at com.intellij.ui.popup.AbstractPopup.lambda$dispose$18(AbstractPopup.java:1503) | at com.intellij.util.ui.EdtInvocationManager.invokeLaterIfNeeded(EdtInvocationManager.java:101) | at com.intellij.ide.IdeEventQueue.ifFocusEventsInTheQueue(IdeEventQueue.java:186) | at com.intellij.ide.IdeEventQueue.executeWhenAllFocusEventsLeftTheQueue(IdeEventQueue.java:140) | at com.intellij.openapi.wm.impl.FocusManagerImpl.doWhenFocusSettlesDown(FocusManagerImpl.java:175) | at com.intellij.openapi.wm.impl.IdeFocusManagerImpl.doWhenFocusSettlesDown(IdeFocusManagerImpl.java:36) | at com.intellij.ui.popup.AbstractPopup.dispose(AbstractPopup.java:1500) | at com.intellij.ui.popup.WizardPopup.dispose(WizardPopup.java:163) | at com.intellij.ui.popup.list.ListPopupImpl.dispose(ListPopupImpl.java:337) | at com.intellij.ui.popup.PopupFactoryImpl$ActionGroupPopup.dispose(PopupFactoryImpl.java:287) | at com.intellij.openapi.util.ObjectTree.runWithTrace(ObjectTree.java:136) | at com.intellij.openapi.util.ObjectTree.executeAll(ObjectTree.java:166) | at com.intellij.openapi.util.Disposer.dispose(Disposer.java:155) | at com.intellij.openapi.util.Disposer.dispose(Disposer.java:143) | at com.intellij.ui.popup.WizardPopup.disposeAllParents(WizardPopup.java:266) | at com.intellij.ui.popup.list.ListPopupImpl.handleNextStep(ListPopupImpl.java:444) | at com.intellij.ui.popup.list.ListPopupImpl._handleSelect(ListPopupImpl.java:416) | at com.intellij.ui.popup.list.ListPopupImpl.handleSelect(ListPopupImpl.java:372) | at com.intellij.ui.popup.PopupFactoryImpl$ActionGroupPopup.handleSelect(PopupFactoryImpl.java:304) | at com.intellij.ui.popup.list.ListPopupImpl$MyMouseListener.mouseReleased(ListPopupImpl.java:629) | at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:298) | at java.desktop/java.awt.Component.processMouseEvent(Component.java:6652) | at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3345) | at com.intellij.ui.popup.list.ListPopupImpl$MyList.processMouseEvent(ListPopupImpl.java:706) | at java.desktop/java.awt.Component.processEvent(Component.java:6417) | at java.desktop/java.awt.Container.processEvent(Container.java:2263) | at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5027) | at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321) | at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859) | at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918) | at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547) | at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488) | at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307) | at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2784) | at java.desktop/java.awt.Component.dispatchEvent(Component.java:4859) | at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778) | at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727) | at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721) | at java.base/java.security.AccessController.doPrivileged(Native Method) | at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) | at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95) | at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751) | at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749) | at java.base/java.security.AccessController.doPrivileged(Native Method) | at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) | at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748) | at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:887) | at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.java:816) | at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:753) | at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$7(IdeEventQueue.java:443) | at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:825) | at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:442) | at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:794) | at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:494) | at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) | at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) | at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) | at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) | at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) | at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90) | 2021-08-28 22:40:07,712 [4906398] ERROR - diff.impl.DiffRequestProcessor - IntelliJ IDEA 2021.2.1 Build #IC-212.5080.55 | 2021-08-28 22:40:07,712 [4906398] ERROR - diff.impl.DiffRequestProcessor - JDK: 11.0.11; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o. | 2021-08-28 22:40:07,712 [4906398] ERROR - diff.impl.DiffRequestProcessor - OS: Linux

[AzureMarker]

What version of the plugin are you using?

[ghost]

this was with 0.2.4, I see you released again I'll try with that and get back to you in the next week or so, but it's unrelated to the replace feature I think. Thanks.

[AzureMarker]

I tried adding 500 lines containing the following and didn't see any issues:

// Test 111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111

Unfortunately the stack traces don't give much info about what might be going wrong in my code. This issue gives some possible reasons for the error: https://github.com/antlr/jetbrains-plugin-sample/issues/9

Did you have any unclosed strings or comments when the error occurred?

[ghost]

I just got the same issue again after upgrading lalrpop plugin. No unclosed strings or comments and I still don't know what triggers it. Play around with the code for a bit though and eventually the comments break. Nothing is a long line. I'll just be typing, eventually cause a syntax error and they break. Other times I cause syntax errors and it won't happen.

Sorry, this one is very hard to catch. It may take weeks before I notice a pattern. It is seemingly random! The wrong offset error still prints though - that is the only consistency.

[AzureMarker]

Do you see the problem when you're typing embedded Rust code or LALRPOP code? Or maybe when you go between them?

[ghost]

"Going between them" is probably a more accurate statement. Certainly just typing comments endlessly doesn't seem to trigger it. It's when I'm coding intensely - a lot of changes occurring rapidly, syntax errors, copying and pasting, etc.

That's why it's so difficult to catch. What I may need to do is record my desktop and convert it to a gif while I trigger the behavior, if you're not able to reproduce. Would that be helpful?

[AzureMarker]

That would be helpful, and at very least help document the behavior.

[ghost]

@AzureMarker it may be a while till I screen record, but I did just activate a .lalrpop file that was previously open and all comments highlighting was lost as soon as I typed:

// TODO: use &str instead of String

This time I got logged:

2021-09-08 17:57:50,257 [14208845] ERROR - enapi.editor.impl.DocumentImpl - com.intellij.openapi.editor.ex.util.LexerEditorHighlighter(com.mdrobnak.lalrpop.LpLexerAdaptor): Error updating after DocumentEventImpl[myOffset=279, myOldLength=0, myNewLength=59].

and then:

2021-09-08 17:57:55,051 [14213639] ERROR - chcopy.TextWithMarkupProcessor - IdeaLoggingEvent[message=Error generating text with markup, throwable=java.lang.AssertionError: Unexpected range returned by highlighter: 43:44, prevEnd: 0, scanned range: 242:338, resulting range: 242:44, highlighter: com.intellij.openapi.editor.ex.util.LexerEditorHighlighter(com.mdrobnak.lalrpop.LpLexerAdaptor): 'use std::str::FromStr;

I'll try to do a recording, but that was the simplest scenario I've witnessed so thought to share. The file had been opened for longer and I can't remember all changes I made prior to typing that TODO. When I do record I'll try to start the recording showing freshly loading intellij.

[AzureMarker]

Thanks, looks like there is an issue in the highlighter. I'll take a closer look soon.

[AzureMarker]

Unexpected range returned by highlighter: 43:44, prevEnd: 0

This makes me think that the Rust code sections might be causing the issue. They are handled specially by the lexer (basically as one big token). I can't find any issue just looking at the code though. Additionally, the highlighting lexer is implemented by JetBrains, so there's not much code on my side to mess up. I'll continue looking into this though, thanks for continuing to share details.

Did this issue start happening recently, or has it always been there?

[ghost]

I am now noticing on 0.2.6 IntelliJ shows a baloon message for "IDE FATAL ERRORS" and an UI appears that would normally allow me to send the error report to the author of the plugin, but the "Report and Clear All" button is grayed out (disabled). Anyway you can enable that feature and then I can send you the error report next time?

[AzureMarker]

That's a good idea. I should be able to look into that this week.

[AzureMarker]

I've started looking at this (error reporting). Actually sending the error report is up to the plugin developer, so I'll need to set up something like Sentry or redirect to GitHub issues. Working on that now.

[AzureMarker]

Added the error reporter, will do a release soon: https://github.com/AzureMarker/intellij-lalrpop/commit/7e6c6d18b7216945354d89ee9bf5a8b295f3fb99

[AzureMarker]

Released 0.2.7 with the error reporter.

[AzureMarker]

FYI I think at least one of these exceptions got submitted to Sentry:

I fixed a few other errors I saw in Sentry, and will get to this one hopefully soon.

[AzureMarker]

I think I found a way to reproduce this error. Start with this:

grammar;

Test: () = "test";

Then quickly add an empty code block like so:

grammar;

Test: () = "test" => {};

The error details will show that it got the exception when analyzing the file right before the second curly brace.

[evbo]

I noticed this behavior a while ago, but seems more stable now in the latest release. Maybe okay to close?

[AzureMarker]

I just tried the reproduction steps I posted, and I still see an error. I investigated it a bunch a while ago, but it's really hard to debug since it is almost all in the internals of IntelliJ's lexer code.

I haven't seen the error pop up on Sentry recently, so maybe it's not seen as much in practice anymore.