search

janino-compiler / janino

Janino is a super-small, super-fast Java™ compiler.
http://janino-compiler.github.io/janino
Other
1.25k stars 208 forks source link

spark compilation failed with ArrayIndexOutOfBoundsException #208

Closed igreenfield closed 9 months ago

igreenfield commented 1 year ago

Caused by: java.lang.ArrayIndexOutOfBoundsException at org.codehaus.janino.UnitCompiler.updateLocalVariableInCurrentStackMap(UnitCompiler.java:13568) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.store(UnitCompiler.java:12758) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.store(UnitCompiler.java:12730) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2742) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.access$2700(UnitCompiler.java:236) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1589) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1575) ~[janino-3.1.9.jar:?] at org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3842) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1575) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1661) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3658) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3329) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1447) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1420) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:829) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1026) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.access$700(UnitCompiler.java:236) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$3.visitMemberClassDeclaration(UnitCompiler.java:425) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$3.visitMemberClassDeclaration(UnitCompiler.java:418) ~[janino-3.1.9.jar:?] at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1533) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:418) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1397) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:864) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:442) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:236) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$3.visitPackageMemberClassDeclaration(UnitCompiler.java:422) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$3.visitPackageMemberClassDeclaration(UnitCompiler.java:418) ~[janino-3.1.9.jar:?] at org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1688) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:418) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:392) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.access$000(UnitCompiler.java:236) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$2.visitCompilationUnit(UnitCompiler.java:363) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler$2.visitCompilationUnit(UnitCompiler.java:361) ~[janino-3.1.9.jar:?] at org.codehaus.janino.Java$CompilationUnit.accept(Java.java:371) ~[janino-3.1.9.jar:?] at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:361) ~[janino-3.1.9.jar:?] at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:264) ~[janino-3.1.9.jar:?] at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:294) ~[janino-3.1.9.jar:?] at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:288) ~[janino-3.1.9.jar:?] at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:267) ~[janino-3.1.9.jar:?] at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:82) ~[commons-compiler-3.1.9.jar:?] at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:1512)

aunkrig commented 1 year ago

Hey Izek,

that is a bit too little information to reproduce the problem... what code are you trying to compile? Please provide a small, runnable example.

CU Arno

aunkrig commented 1 year ago

Ping.

aunkrig commented 12 months ago

Hi Izek,

I cannot compile and run your example without installing and understanding Spark, which I am not planning to do. Can you please provide a stand-alone example that depends solely on Janino?

CU Arno

igreenfield commented 12 months ago

janino_issue.zip I attached the full project. @aunkrig OK?

aunkrig commented 11 months ago

Problem reproduced. Boils down to compiling this minimal class body:

class SpecificMutableProjection {

    private void apply_0_7() {

        do {
            boolean isN_50 = false;
            long l = 33;
            continue;
        } while (this == null); // Doesn't matter if condition has constant value or not.

        int val_53 = 9;
        boolean isN_52 = false;
        long val_52 = -1L;         // <== java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -1 is negative
    }
}
aunkrig commented 11 months ago

Seems like when Janino compiles "do { ... continue; } while (false);", it doesn't clean up the StackMap properly. Will look into it.

Workaround: Generate a labeled statement instead of the awkward DO...WHILE non-loop.

igreenfield commented 11 months ago

@aunkrig When do you think there will be a fix for this?

igreenfield commented 11 months ago

@aunkrig I convert DO...WHILE into label and still get the error. this is the new code:


return new GeneratedIteratorForCodegenStage1(refs);
}

/*wsc_codegenStageId*/
final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
private Object[] refs;
private scala.collection.Iterator[] inputs;
private scala.collection.Iterator inadp_input_0;
private boolean prj_prj_isN_0_0;
private boolean prj_prj_isN_5_0;
private boolean prj_prj_isN_8_0;
private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] prj_mStArr_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2];

public GeneratedIteratorForCodegenStage1(Object[] refs) {
this.refs = refs;
}

public void init(int index, scala.collection.Iterator[] inputs) {
partitionIndex = index;
this.inputs = inputs;
inadp_input_0 = inputs[0];

prj_mStArr_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(5, 96);
prj_mStArr_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(7, 160);

}

protected void processNext() throws java.io.IOException {
while ( inadp_input_0.hasNext()) {
InternalRow inadp_row_0 = (InternalRow) inadp_input_0.next();

// common sub-expressions

prj_prj_isN_0_0 = true;
UTF8String prj_val_0 = null;
prj_lbl__0: {
boolean inadp_isN_0 = inadp_row_0.isNullAt(0);
UTF8String inadp_val_0 = inadp_isN_0 ?
null : (inadp_row_0.getUTF8String(0));
if (!inadp_isN_0) {
prj_prj_isN_0_0 = false;
prj_val_0 = inadp_val_0;
break prj_lbl__0;
}

prj_prj_isN_0_0 = false;
prj_val_0 = ((UTF8String) refs[0] /* literal */);
break prj_lbl__0;

}

// common sub-expressions

boolean inadp_isN_1 = inadp_row_0.isNullAt(1);
double inadp_val_1 = inadp_isN_1 ?
-1.0 : (inadp_row_0.getDouble(1));
boolean inadp_isN_2 = inadp_row_0.isNullAt(2);
double inadp_val_2 = inadp_isN_2 ?
-1.0 : (inadp_row_0.getDouble(2));
prj_prj_isN_5_0 = true;
UTF8String prj_val_5 = null;
prj_lbl__1: {
boolean inadp_isN_3 = inadp_row_0.isNullAt(3);
UTF8String inadp_val_3 = inadp_isN_3 ?
null : (inadp_row_0.getUTF8String(3));
if (!inadp_isN_3) {
prj_prj_isN_5_0 = false;
prj_val_5 = inadp_val_3;
break prj_lbl__1;
}

prj_prj_isN_5_0 = false;
prj_val_5 = ((UTF8String) refs[1] /* literal */);
break prj_lbl__1;

}
prj_prj_isN_8_0 = true;
UTF8String prj_val_8 = null;
prj_lbl__2: {
boolean inadp_isN_4 = inadp_row_0.isNullAt(4);
UTF8String inadp_val_4 = inadp_isN_4 ?
null : (inadp_row_0.getUTF8String(4));
if (!inadp_isN_4) {
prj_prj_isN_8_0 = false;
prj_val_8 = inadp_val_4;
break prj_lbl__2;
}

prj_prj_isN_8_0 = false;
prj_val_8 = ((UTF8String) refs[2] /* literal */);
break prj_lbl__2;

}
prj_mStArr_0[1].reset();

prj_mStArr_0[1].zeroOutNullBytes();

prj_mStArr_0[1].write(0, prj_val_0);

if (inadp_isN_1) {
prj_mStArr_0[1].setNullAt(1);
} else {
prj_mStArr_0[1].write(1, inadp_val_1);
}

if (inadp_isN_2) {
prj_mStArr_0[1].setNullAt(2);
} else {
prj_mStArr_0[1].write(2, inadp_val_2);
}

prj_mStArr_0[1].write(3, prj_val_5);

prj_mStArr_0[1].write(4, prj_val_8);

prj_mStArr_0[1].write(5, prj_val_0);

prj_mStArr_0[1].write(6, prj_val_0);
append((prj_mStArr_0[1].getRow()));
if (shouldStop()) return;
}
}

}```
igreenfield commented 11 months ago

@aunkrig Why it is still tagged as waiting for user ?

aunkrig commented 11 months ago

How is the "new code" related to the old code?

And again: Can you please provide a stand-alone example that depends solely on Janino? I understand that you want your problem be solved in your environment, but someone has to isolate and minimize it, and I think that should be you.

igreenfield commented 11 months ago
  1. the new code is with your workaround, using labels and not DO...WHILE.
  2. if you paste the code in the project I sent you it will be compiled.
  3. What about the first issue with the DO...WHILE?
aunkrig commented 11 months ago

I believe I fixed it. Please test.

igreenfield commented 11 months ago

@aunkrig for my test it solves the issue, so please release a new version. thanks.

aunkrig commented 11 months ago

🕺🕺🕺

igreenfield commented 10 months ago

@aunkrig Can you please release a new version with this fix? it is very urgent for me. thanks.

aunkrig commented 10 months ago

Version 3.1.11 is out the door! Please test.

aunkrig commented 9 months ago

Please test.

igreenfield commented 9 months ago

it is good, thanks.