Error in JDT Core during AST creation due to spurious recovery of String template expression by DiagnoseParser

[Madjosz]

With the following code three messages in the Error Log View pop up

package mwe;

import java.util.List;
import java.util.function.Predicate;
import java.util.stream.IntStream;

public class CompilationUnitConversion {
    private Predicate<String> matches() {
        return s -> {
            List<Object> results = List.of();
            return IntStream.range(0, 10).allMatch(results.get(i).);
        };
    }
}

Exception occurred during compilation unit conversion:

Error in JDT Core during AST creation

Problems occurred when invoking code from plug-in: "org.eclipse.jdt.core.manipulation".

java.lang.IllegalArgumentException
    at org.eclipse.jdt.core.dom.ASTNode$NodeList.add(ASTNode.java:1619)
    at java.base/java.util.AbstractList.add(AbstractList.java:111)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2528)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2046)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3021)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3092)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:1370)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2607)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2055)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3021)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3092)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:738)
    at org.eclipse.jdt.core.dom.ASTConverter.buildBodyDeclarations(ASTConverter.java:208)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3439)
    at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:1521)
    at org.eclipse.jdt.core.dom.CompilationUnitResolver.convert(CompilationUnitResolver.java:344)
    at org.eclipse.jdt.core.dom.ASTParser.internalCreateASTCached(ASTParser.java:1269)
    at org.eclipse.jdt.core.dom.ASTParser.lambda$0(ASTParser.java:1120)
    at org.eclipse.jdt.internal.core.JavaModelManager.cacheZipFiles(JavaModelManager.java:5812)
    at org.eclipse.jdt.core.dom.ASTParser.internalCreateAST(ASTParser.java:1120)
    at org.eclipse.jdt.core.dom.ASTParser.createAST(ASTParser.java:868)
    at org.eclipse.jdt.core.manipulation.CoreASTProvider$1.run(CoreASTProvider.java:294)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
    at org.eclipse.jdt.core.manipulation.CoreASTProvider.createAST(CoreASTProvider.java:286)
    at org.eclipse.jdt.core.manipulation.CoreASTProvider.getAST(CoreASTProvider.java:199)
    at org.eclipse.jdt.core.manipulation.SharedASTProviderCore.getAST(SharedASTProviderCore.java:138)
    at org.eclipse.jdt.internal.ui.viewsupport.SelectionListenerWithASTManager$PartListenerGroup.calculateASTandInform(SelectionListenerWithASTManager.java:166)
    at org.eclipse.jdt.internal.ui.viewsupport.SelectionListenerWithASTManager$PartListenerGroup$1.run(SelectionListenerWithASTManager.java:151)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

It also messes with postfix completions on the trailing . after the get() call:

The 'org.eclipse.jdt.ui.postfixCompletion' proposal computer from the 'org.eclipse.jdt.ui' plug-in did not complete normally. The extension has thrown a runtime exception.

System information

Eclipse SDK Version: 2024-03 (4.31) Build id: I20231207-1800

openjdk 21 2023-09-19 LTS OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS) OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

[srikanth-sankaran]

Interesting.

The parse tree for the method comes out to be:

private Predicate<String> matches() {
  return (<no type> s) -> {
  List<Object> results = List.of();
  return IntStream.range(0, 10).allMatch(results.get(i)."");
};
}

See the "" at the end of the return statement inside the lambda!

Basically we have parsed a string template expression where none existed. I suspect DiagnoseParser's black magic having a finger in the pie

[srikanth-sankaran]

Yep. DiagnoseParser inserts a "" at the syntax error site and then during recovery we see a String template. This itself is not the problem per se - org.eclipse.jdt.core.dom.ASTConverter.convert(Expression) is lacking a case for StringTemplates and returns null which then results in IAE.

We may want to steer the DiagnoseParser from inserting "" into the Recovery Scanner's input stream - otherwise code completion may break as reported above

[Madjosz]

Latest build of Eclipse (I20240205-1800) does not throw anymore and shows

Syntax error on token ".", StringLiteral expected after this token

when hovering the last . and on results.get(i) it shows

String Template is a preview feature and disabled by default. Use --enable-preview to enable

This is better (as it does not hard-fail the build) but not at all what this code is about. Content assist also suggests List methods after last . but that might be due to the missing declaration of the i variable and thus the compiler not being able to find a suitable method.

[Madjosz]

aa1a02d82b31a8ecfdd7aed71ecd79178299735e fixed the errors appearing in the Error Log view and I would consider this issue fixed. The spurious recovery of String template might be worth its own issue but there might already be enough issues about String templates so we could revisit this behaviour when the implementation is more mature.