Error during async editor loading: Discontinuous sequence of tokens is generated by lexer

[jgjl]

Hi,

this preprocessor code (to try, just paste into any P4 file):

#elif defined(ABCDEFGHIJKLMNOPQRS)

crashes the plugin with this message:

com.intellij.diagnostic.PluginException: Discontinuous sequence of tokens is generated by lexer: org.antlr.intellij.adaptor.lexer.ANTLRLexerAdaptor [Plugin: org.p4.p4plugin]
    at com.intellij.ide.plugins.PluginManagerCore.createPluginException(PluginManagerCore.java:274)
    at com.intellij.diagnostic.PluginProblemReporterImpl.createPluginExceptionByClass(PluginProblemReporterImpl.java:12)
    at com.intellij.diagnostic.PluginException.createByClass(PluginException.java:91)
    at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.throwException(ValidatingLexerWrapper.java:126)
    at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.advance(ValidatingLexerWrapper.java:94)
    at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.doSetText(LexerEditorHighlighter.java:453)
    at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.setText(LexerEditorHighlighter.java:420)
    at com.intellij.openapi.editor.ex.util.LayeredLexerEditorHighlighter.setText(LayeredLexerEditorHighlighter.java:124)
    at com.intellij.openapi.fileEditor.impl.text.TextEditorImpl.loadEditorInBackground(TextEditorImpl.java:74)
    at com.intellij.openapi.fileEditor.impl.text.PsiAwareTextEditorImpl.loadEditorInBackground(PsiAwareTextEditorImpl.java:42)
    at com.intellij.openapi.fileEditor.impl.text.AsyncEditorLoader.lambda$scheduleLoading$0(AsyncEditorLoader.java:95)
    at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:825)
    at com.intellij.openapi.fileEditor.impl.text.AsyncEditorLoader.lambda$scheduleLoading$1(AsyncEditorLoader.java:93)
    at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.insideReadAction(NonBlockingReadActionImpl.java:521)
    at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$attemptComputation$3(NonBlockingReadActionImpl.java:486)
    at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1078)
    at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runInReadActionWithWriteActionPriority$0(ProgressIndicatorUtils.java:78)
    at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runActionAndCancelBeforeWrite(ProgressIndicatorUtils.java:161)
    at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runWithWriteActionPriority$1(ProgressIndicatorUtils.java:118)
    at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:57)
    at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:183)
    at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:705)
    at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:647)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:63)
    at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:170)
    at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:57)
    at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runWithWriteActionPriority(ProgressIndicatorUtils.java:115)
    at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runInReadActionWithWriteActionPriority(ProgressIndicatorUtils.java:78)
    at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.attemptComputation(NonBlockingReadActionImpl.java:486)
    at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$transferToBgThread$1(NonBlockingReadActionImpl.java:407)
    at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:216)
    at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:27)
    at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:195)
    at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:213)
    at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:184)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
    at java.base/java.lang.Thread.run(Thread.java:829)

I had a look at the ANTLR grammar already, but did not immediately see anything related to this issue. I will keep looking, but if anyone has an idea: input is welcome!

[jgjl]

I found the issue. The ANTLR spec uses elseif instead of elif as specified in the P4 Lang spec: 6.2. Preprocessing

[jgjl]

Another pattern that the parser cannot handle seems to be multi-line preprocessor code:

#define TEST(param1) \
    state ctrl_##param1 { \
}