java.lang.ClassCastException: antlr.CommonHiddenStreamToken cannot be cast to antlr.Token

[MCMicS]

With the last updates of Idea and the plugin I get following error. If I press reload config it work for an short time but then it appears again. OS: Windows IDEA: 2019.1.3

org.infernus.idea.checkstyle.exception.CheckstyleToolException: Exception was thrown while processing Foo.java
    at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.executeCommand(CheckstyleActionsImpl.java:136)
    at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.scan(CheckstyleActionsImpl.java:76)
    at org.infernus.idea.checkstyle.checker.CheckStyleChecker.scan(CheckStyleChecker.java:45)
    at org.infernus.idea.checkstyle.checker.ScanFiles.lambda$checkFiles$6(ScanFiles.java:190)
    at java.util.Optional.map(Optional.java:215)
    at org.infernus.idea.checkstyle.checker.ScanFiles.checkFiles(ScanFiles.java:190)
    at org.infernus.idea.checkstyle.checker.ScanFiles.processFilesForModuleInfoAndScan(ScanFiles.java:149)
    at org.infernus.idea.checkstyle.checker.ScanFiles.call(ScanFiles.java:75)
    at org.infernus.idea.checkstyle.checker.ScanFiles.call(ScanFiles.java:31)
    at com.intellij.openapi.application.impl.ApplicationImpl$2.call(ApplicationImpl.java:339)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.puppycrawl.tools.checkstyle.api.CheckstyleException: Exception was thrown while processing Foo.java
    at com.puppycrawl.tools.checkstyle.Checker.processFiles(Checker.java:307)
    at com.puppycrawl.tools.checkstyle.Checker.process(Checker.java:220)
    at org.infernus.idea.checkstyle.service.cmd.OpScan.processAndAudit(OpScan.java:76)
    at org.infernus.idea.checkstyle.service.cmd.OpScan.execute(OpScan.java:52)
    at org.infernus.idea.checkstyle.service.cmd.OpScan.execute(OpScan.java:22)
    at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.executeCommand(CheckstyleActionsImpl.java:130)
    ... 13 more
Caused by: java.lang.ClassCastException: antlr.CommonHiddenStreamToken cannot be cast to antlr.Token
    at antlr.CharScanner.makeToken(CharScanner.java:173)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.mBLOCK_COMMENT_CONTENT(GeneratedJavaLexer.java:1258)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.mBLOCK_COMMENT_BEGIN(GeneratedJavaLexer.java:1199)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.nextToken(GeneratedJavaLexer.java:335)
    at antlr.TokenStreamHiddenTokenFilter.consume(TokenStreamHiddenTokenFilter.java:38)
    at antlr.TokenStreamHiddenTokenFilter.consumeFirst(TokenStreamHiddenTokenFilter.java:42)
    at antlr.TokenStreamHiddenTokenFilter.nextToken(TokenStreamHiddenTokenFilter.java:122)
    at antlr.TokenBuffer.fill(TokenBuffer.java:69)
    at antlr.TokenBuffer.LA(TokenBuffer.java:80)
    at antlr.LLkParser.LA(LLkParser.java:52)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaRecognizer.compilationUnit(GeneratedJavaRecognizer.java:158)
    at com.puppycrawl.tools.checkstyle.JavaParser.parse(JavaParser.java:97)
    at com.puppycrawl.tools.checkstyle.TreeWalker.processFiltered(TreeWalker.java:160)
    at com.puppycrawl.tools.checkstyle.api.AbstractFileSetCheck.process(AbstractFileSetCheck.java:85)
    at com.puppycrawl.tools.checkstyle.Checker.processFile(Checker.java:332)
    at com.puppycrawl.tools.checkstyle.Checker.processFiles(Checker.java:294)
    ... 18 more

[jshiell]

Does this happen with Checkstyle on the command line as well? That exception does look like it's coming out of the parser in Checkstyle rather than from the plugin.

[MCMicS]

not tested yet. but we have a job on buildsystem to use checkstyle as commandline and there are no errors. Maybe it is some library in IDE classpath which causes the conflict

[jshiell]

It wouldn't be the first time that's happened. I certainly can't find any relevant bug in Checkstyle's list.

Could you please attach your rules file? I'll try have a look at this ASAP, although it'll probably be this weekend at the earliest. I don't have a Windows box either, but it doesn't look like a platform issue 🤞

[MCMicS]

thanks for look.

rules.zip

I've take a look on the release notes of checkstyle and see following: 8.20: Remove usage of antlr specific methods from our checks. Author: rnveach #6624 8.21: API: make api not depend on antlr by splitting DetailAST between implementation and interface. Author: rnveach #3417

Maybe one of them is causing it

I choosed 8.19 within settings but error still occurs.

[jshiell]

Stupid thought - did this happen while you were editing a file, e.g. might the file have been syntacticly incorrect at the time?

Checkstyle only commit to supporting syntactically correct files - the output for anything unparsable is unpredictable, by design (i.e. it just propagates whatever happens). As such, plugin authors need to do some extra error handling to catch these cases, and every so often a new variation pops up that breaks this. Hence I'm wondering if this might be one of those cases.

[MCMicS]

Yes most of them occurs if I edit a file. And IDea saves automatically and an check could be run in background. But then the message is wrong:

java.lang.ClassCastException: antlr.CommonHiddenStreamToken cannot be cast to antlr.Token

[jshiell]

Smashing, thanks - hopefully a fix shouldn't be too hard in this case.

[MCMicS]

could you reproduce it while editing files?

[jshiell]

Haven't tried as yet, this was just from thinking through the problem on my commute.

[MCMicS]

I thought this is not the main problem. Now I get the message too if I change nothing but switching between files. Sorry for that bad news

[jshiell]

All is not yet lost - it's definitely hitting this case, having traced it, although it may be hitting another case as well.

And it's quick to fix, so I can get this out now (5.29.1) and - assuming problems continue - can investigate further when I've more time to hand.

[MCMicS]

ok thanks. I will try it as soon as possible

[jshiell]

Thanks - no rush, mind - looks like the weekend is the earliest I'll get time to have another go 😄

[MCMicS]

I've updated the plugin but a new error occurs. But this is not happen so many times like before

2019-07-11 17:31:38,534 [27599292] WARN - a.checkstyle.checker.ScanFiles - An error occurred while scanning a file.

org.infernus.idea.checkstyle.exception.CheckStylePluginParseException: Exception was thrown while processing Foo.java
    at org.infernus.idea.checkstyle.service.ExceptionWrapper.wrap(ExceptionWrapper.java:34)
    at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.executeCommand(CheckstyleActionsImpl.java:132)
    at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.scan(CheckstyleActionsImpl.java:76)
    at org.infernus.idea.checkstyle.checker.CheckStyleChecker.scan(CheckStyleChecker.java:45)
    at org.infernus.idea.checkstyle.checker.ScanFiles.lambda$checkFiles$6(ScanFiles.java:190)
    at java.util.Optional.map(Optional.java:215)
    at org.infernus.idea.checkstyle.checker.ScanFiles.checkFiles(ScanFiles.java:190)
    at org.infernus.idea.checkstyle.checker.ScanFiles.processFilesForModuleInfoAndScan(ScanFiles.java:149)
    at org.infernus.idea.checkstyle.checker.ScanFiles.call(ScanFiles.java:75)
    at org.infernus.idea.checkstyle.checker.ScanFiles.call(ScanFiles.java:31)
    at com.intellij.openapi.application.impl.ApplicationImpl$2.call(ApplicationImpl.java:339)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.puppycrawl.tools.checkstyle.api.CheckstyleException: Exception was thrown while processing Foo.java
    at com.puppycrawl.tools.checkstyle.Checker.processFiles(Checker.java:307)
    at com.puppycrawl.tools.checkstyle.Checker.process(Checker.java:220)
    at org.infernus.idea.checkstyle.service.cmd.OpScan.processAndAudit(OpScan.java:76)
    at org.infernus.idea.checkstyle.service.cmd.OpScan.execute(OpScan.java:52)
    at org.infernus.idea.checkstyle.service.cmd.OpScan.execute(OpScan.java:22)
    at org.infernus.idea.checkstyle.service.CheckstyleActionsImpl.executeCommand(CheckstyleActionsImpl.java:130)
    ... 13 more
Caused by: java.lang.ClassCastException: antlr.CommonHiddenStreamToken cannot be cast to antlr.Token
    at antlr.CharScanner.makeToken(CharScanner.java:173)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.mBLOCK_COMMENT_CONTENT(GeneratedJavaLexer.java:1258)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.mBLOCK_COMMENT_BEGIN(GeneratedJavaLexer.java:1199)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.nextToken(GeneratedJavaLexer.java:335)
    at antlr.TokenStreamHiddenTokenFilter.consume(TokenStreamHiddenTokenFilter.java:38)
    at antlr.TokenStreamHiddenTokenFilter.consumeFirst(TokenStreamHiddenTokenFilter.java:42)
    at antlr.TokenStreamHiddenTokenFilter.nextToken(TokenStreamHiddenTokenFilter.java:122)
    at antlr.TokenBuffer.fill(TokenBuffer.java:69)
    at antlr.TokenBuffer.LA(TokenBuffer.java:80)
    at antlr.LLkParser.LA(LLkParser.java:52)
    at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaRecognizer.compilationUnit(GeneratedJavaRecognizer.java:158)
    at com.puppycrawl.tools.checkstyle.JavaParser.parse(JavaParser.java:97)
    at com.puppycrawl.tools.checkstyle.TreeWalker.processFiltered(TreeWalker.java:160)
    at com.puppycrawl.tools.checkstyle.api.AbstractFileSetCheck.process(AbstractFileSetCheck.java:85)
    at com.puppycrawl.tools.checkstyle.Checker.processFile(Checker.java:332)
    at com.puppycrawl.tools.checkstyle.Checker.processFiles(Checker.java:294)
    ... 18 more

[jshiell]

Where are you seeing the exception? That looks like the log file?

An exception will be logged - we've received an error from the plugin, and have no definite way of differentiating an exception caused by a programming error (e.g. the plugin or Checkstyle has broken) and one caused by unparsable input. Hence we'd log it, and it should be present in the event log and in the IDEA log files - but it shouldn't cause an uncaught exception (i.e. the flashing red icon in the bottom right of the IDE window).

The alternative would be to hide it, which would a) hide real errors and b) make it looks like the scan wasn't happening at all.

[MCMicS]

the log appears in error log within ide (bottom right red balloon)

[jshiell]

Hmm. Are you able to share a sample of code which produces this? I've managed to produce similar traces, but haven't managed the ClassCastException exactly yet.

I appreciate this may not be possible for confidentiality/commercial reasons, of course.

[MCMicS]

I can try to reproduce it with sample code but the original code (commercial) cannot be shared it occurs in many files. After your fix mostly before commit to SVN (with check with checkstyle)

[jshiell]

That's perfect - that's what I was missing. I was presuming it was occurring with the real-time scan, as parse problems tend to be linked to code being edited. It doesn't look like we're handling this case for on-commit scans, as we expect the code to be parseable.

Which raises the question of what to do with this. Checkstyle itself is throwing the error, so we have a choice of either continuing to raise it as an error, or of suppressing it on the basis that the user knows what they're doing. What do you think?

[jshiell]

And there's a third option, which I'm working on now - which is to clean up the error output so it's clear as to the cause, e.g.

[jshiell]

Error recording changes have been released in 5.29.2.

[MCMicS]

Thanks I will try it monday on work

But to clarify: The first error " antlr.CommonHiddenStreamToken cannot be cast to antlr.Token" occurs during runtime checks if editing files or switching editors with valid files. After 5.29.1 the error mostly occurs during pre commit check but sometimes at runtime.

I use OpenJDK 11 and java11 sources. Maybe this is another hint.

[MCMicS]

Tested the new version on work. The error within IDE Event Log occurs if I change a file and the file is checked (either during runtime or by click 'Check Current Change List'.

17:28   The source file could not be parsed by Checkstyle
            The scan failed due to a parse error: Exception was thrown while processing Foo.java
            Root cause:
            antlr.CommonHiddenStreamToken cannot be cast to antlr.Token
            java.lang.ClassCastException: antlr.CommonHiddenStreamToken cannot be cast to antlr.Token

            at antlr.CharScanner.makeToken(CharScanner.java:173)

            at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.mBLOCK_COMMENT_CONTENT(GeneratedJavaLexer.java:1258)

            at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.mBLOCK_COMMENT_BEGIN(GeneratedJavaLexer.java:1199)

            at com.puppycrawl.tools.checkstyle.grammar.GeneratedJavaLexer.nextToken(GeneratedJavaLexer.java:335)

            at antlr.TokenStreamHiddenTokenFilter.consume(TokenStreamHiddenTokenFilter.java:38)

The Checkstyle Window the show that the file is unparsable for a time. If I click ''Check Current Change List' again, the check may works then. It seems that the Check is not possible for a while after change file content. (may be until Idea has build index for changed content)

[jshiell]

Hmm. I've made no progress on this. The file logic is consistent for the checks - for each file we check if it is either not yet on the disk, or unsaved and, if so, we create a temporary file and scan that. Otherwise we scan using the file on disk. Either way, this logic is run before the scan takes place (in ScannableFile), so there should be no time lag.

Purely speaking in theory it could be related to indexing - we rely on IDEA's API to get the file metadata and decide how to scan it, but it seems odd.

[MCMicS]

ok I'll try to create a simple test file with small project. I've tested it with default google check rules and still occurs. So it is not an rule config problem. Today I got the error if i click check files for package. And after 20 times the check works. It seems like random.

Maybe I can reproduce it with simple test case

[jshiell]

Thank you. I fear my environment may not be a good match for this, as I've only small projects to work with, and an entirely different OS with a different filesystem and approach to file locking.

🤞

[MCMicS]

Is there anyway to enable debug logging for your plugin to help for looking the cause?

[jshiell]

The only way I'm aware of is to clone the project and run the runIdea target - this will start up a version of IDEA 2017 CE, which is the API we compile against, in the debugger and allow you to go mad with breakpoints. It's a bit of a hassle tbh, but very useful.

[MCMicS]

Thanks I will try it if can reproduce it. But if seem to work with versions before 2019.1.2 Maybe its a version problem but we will see if if reproduce it.

But this will not happen before monday. But then i will try it

[jshiell]

Hmm, odd if it does work before 2019.1.2 - there's nothing I'm aware of that should change behaviour in that case.

[MCMicS]

CheckStyle_Error.zip

I have created a simple sample. May be occurs only if one rule exists with properties. Without the rule the error not occurs so easy. I uploaded a zip (pwd: checkstyle). Maybe one of the plugins interfers with yours.

My Version: IDEA 2019.1.3 Build #-IU-191.7479.19

Steps to reproduce:

1. Checkout Project
2. Add rule within src/main/resources
3. set property to project root
4. Goto SampleValue class and remove private long second;
5. click Check Current file (may be two or three time)

Github: https://github.com/MCMicS/CheckStyle_Error

Tested it with 2018.3:

• Checkstyle Plugin 5.24.3: I can't reproduce the error. (Checkstyle version 8.16)
• Checkstyle Plugin 5.30: I can't reproduce the error. (Checkstyle version 8.22)

It seem to be with the 2019.1.3 version

Update my 2019.1.3 to latest plugin version and have a look if the error still occurs

[jshiell]

Thanks - I'll try have a poke at this this weekend.

[jshiell]

I made no progress on this during the weekend I fear, as events intervened, but it's top of my todo list for plugin work.

[MCMicS]

Hi. No Problem.

[jshiell]

Bad news - I can't reproduce on either 2019.1.3 or 2019.2. I can happily spam the scan current file button and it seems fine. So either I'm going very wrong, or this is an issue specific to Windows (which wouldn't be entirely surprising, given how different the IO stacks are).

Does it still occur for your on 2019.2?

I'm also going trying updating the API we build against from 2017 to 2018, just in case there were any relevant deprecations.

[MCMicS]

I've last week and now I am at holiday for some days. I will have a look with the 2019.2 update.

But this could be done in about 2 weeks

[jshiell]

No rush - I hope you have a good holiday!

[MCMicS]

Hi, I've tested it with 2019.2 and the error still occurs :(

So it's maybe a Windows specific problem.

[MCMicS]

Got a better message now. Maybe it help to find a solution.

The scan failed due to a parse error: Exception was thrown while processing Foo.java
            Root cause:
            class antlr.CommonHiddenStreamToken cannot be cast to class antlr.Token (antlr.CommonHiddenStreamToken is in unnamed module of loader com.intellij.ide.plugins.cl.PluginClassLoader @56d9833e; antlr.Token is in unnamed module of loader org.infernus.idea.checkstyle.util.ChildFirstURLClassLoader @443f2233)
            java.lang.ClassCastException: class antlr.CommonHiddenStreamToken cannot be cast to class antlr.Token (antlr.CommonHiddenStreamToken is in unnamed module of loader com.intellij.ide.plugins.cl.PluginClassLoader @56d9833e; antlr.Token is in unnamed module of loader org.infernus.idea.checkstyle.util.ChildFirstURLClassLoader @443f2233)

            at antlr.CharScanner.makeToken(CharScanner.java:173)

[jshiell]

I fear it doesn't tell us a lot more, other than module info for Java 9+. The ChildFirstURLClassLoader is actually our main line of defence against class path issues - it ensures we check the local (i.e. Checkstyle) class path before delegated to the IDEA class path (which does have conflicting versions of some dependencies).

I'm kind of at a dead end tbh, given I can't reproduce it, and given that IDEA 2018.2.x works fine with the same code. I didn't spot anything even vaguely relevant when updating to the 2018 SDK, and I can't justify an upgrade to the 2019 SDK at present (as that will cut off users using IDEA 2018 - and I suspect it's probably a red herring anyway).

I did have a quick poke around https://youtrack.jetbrains.com but can't spot anything relevant. Mind, I've never had much searching on that thing.

The only thought I had is that it's odd for an issue like this to only be seen by one person, which does make me wonder if there's something unusual in your setup which is changing things. Does the problem still occur if you disable all other third-party plugins, for instance? And it is Windows 10 you're using, I presume?

[MCMicS]

Ok. I can try a clean Installation with checkstyle plugin only

[MCMicS]

I had installed another checkstyle plugin (QAPlug). Maybe this was the cause. I've removed it and have a look if the error is gone.

[MCMicS]

The errot not occurs for now. So I thought it was the other plugin. I close this issue.

[MCMicS]

Now I got them again after check from the commit dialog. Then it comes on normal usage too.

[jshiell]

To be fair it would have been a bit odd, as IDEA is fairly consistent on classpath isolation.

[MCMicS]

I've looked in my other plugin and only the PMD plugin contains additional antlr libs. I try to remove this and try it again. I know it seems odd. But yesterday the error occurs only after commit to SVN.

[zielint0]

I had this bug multiple times on 5.34.0. After IDEA restart it is gone.

[MCMicS]

After Restart the errors are gone but they occurs after time of working with IDE.

Tested with latest Checkstyle Plugin and IDEA 2019.2.4

[jshiell]

This is generally (but not always, as the history shows) caused by changes in how the core Checkstyle library handles (or doesn't handle) errors. Can you please tell me which version of Checkstyle you are using, and also provide a stack trace from the errors?

[MCMicS]

I can try to get the stacktrace. Because the error is non-reproducible it is difficult to get the stack. But after long work time with IDE the error occurs.

I've tested it with Checkstyle 8.22 and 8.26