GPF when lombok is used

[sudhirtumati]

Java -version output

openjdk version "12.0.2" 2019-07-16 OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.2+10) Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.15.1, JRE 12 Windows 10 amd64-64-Bit Compressed References 20190718_139 (JIT enabled, AOT enabled) OpenJ9 - 0f66c6431 OMR - ec782f26 JCL - 06c2cc3322 based on jdk-12.0.2+10)

Summary of problem

Not able to compile code when lombok annotations are present

Diagnostic files

diagnostic-files.zip Core dump

[GaborSzabo7]

Hi @sudhirtumati

I found a similiar issue. Is it completely same or should I open a new bug? https://stackoverflow.com/questions/58274481/maven-compile-failure-using-openj9-with-lombok app.zip

[DanHeidinga]

FYI @andrewcraik - can you get someone to take a look at this? The StackOverflow link is a crash in the jit dll on v0.16.0 and has a test case to reproduce it.

[andrewcraik]

@liqunl can you take a look at this failure?

[liqunl]

@sudhirtumati @GaborSzabo7 Could you kindly provide instructions on how to set up the machine to run the test and reproduce the issue?

[GaborSzabo7]

@liqunl I installed Eclipse OpenJ9 VM AdoptOpenJDK in my machine and after that I created a very simple Spring Boot application.

Instructions:

1. Download app.zip (you can find it in previous comment)
2. Open terminal
3. Browse folder you can find pom.xml of app.zip
4. Execute mvn clean compile

[liqunl]

The problem is that an reference field contains a bad value

> !j9object 0x00000007FFA0B290
!J9Object 0x00000007FFA0B290 {
        struct J9Class* clazz = !j9class 0x1DBB600 // java/lang/reflect/Field
        Object flags = 0x00000000;
        I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
        Z override = 0x00000000 (offset = 8) (java/lang/reflect/AccessibleObject)
        Ljava/lang/Object; accessCheckCache = !fj9object 0x1 (offset = 4) (java/lang/reflect/AccessibleObject)

accessCheckCache has value 1. The only place that sets the value for this field is AccessibleObject.slowVerifyAccess.

I ran the test with -Xint, the crash occurs in libj9vm29.so. From the core dump I found an object with the same problem. FYI @DanHeidinga

[DanHeidinga]

@liqunl Can you dump the core file on the internal network at /team/triage/6933? It'll help speed the investigation if we can start from your setup

[liqunl]

@DanHeidinga I don't have permission to create a folder at /team/triage, please find the core dump at /team/triage/liqunl/openj9-6933

[andrewcraik]

@DanHeidinga this seems to be one you were looking at / having someone how works on the VM take a look at - do you know what the status is given this is tagged for 0.18? Do you need anything else from the JIT side?

[pshipton]

The investigation on this hasn't completed and it's getting to be too late to add more fixes, moving out to the next release.

[pshipton]

@DanHeidinga is there somebody who can take a look at this one?

[DanHeidinga]

@fengxue-IS Can you take a look at this?

[fengxue-IS]

looking into this

[pshipton]

Moving forward since I don't see any fix in hand and it's becoming too late to update the 0.19 release. Note the branch for the 0.20 release occurs on March 8.

[pshipton]

@fengxue-IS any updates on this?

[pshipton]

We're past Milestone 2 for the 0.20.0 release, moving this forward to the next release.

[babsingh]

You can easily reproduce the issue through javac; instead of the above maven application. I derived the below steps by running the maven application in debug mode (-X).

Easy Steps

1. Get https://projectlombok.org/downloads/lombok.jar
2. Create HelloWorld.java

   public class HelloWorld {
   public static void main(String[] args) {
       System.out.println("Hello, Lombak");
   }
   }

3. Get OpenJ9 JDK14 from AdoptOpenJDK
4. javac -cp lombok-1.18.12.jar HelloWorld.java should reproduce the reported failure

Failure and Debug Details

1. The SIGSEGV happens because https://projectlombok.org/ (Github: https://github.com/rzwitserloot/lombok) installs a custom annotation processor, Processor lombok.launch.AnnotationProcessorHider$AnnotationProcessor
2. Get debug info during javac with -verbose: javac -cp lombok-1.18.12.jar -verbose HelloWorld.java
3. Another note ... the failure does not happen with OpenJ9 JDK8 and JDK11 ... it only happens with OpenJ9 JDK13 and JDK14.

Potential Temporary Fix

Disabling annotation processors with -proc:none resolves the javac failure. But, this may also disable Lombok features.

[BAD] JDK14 failure happens in Round 1 of annotation processing (javac -verbose)

Round 1:
    input files: {HelloWorld}
    annotations: []
    last round: false
Unhandled exception
Type=Segmentation error vmState=0x00000000

[GOOD] JDK8 behavior (javac -verbose)

Round 1:
    input files: {HelloWorld}
    annotations: []
    last round: false
Processor lombok.launch.AnnotationProcessorHider$AnnotationProcessor matches [] and returns false.
[loading ZipFileIndexFileObject[/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class)]]
[loading ZipFileIndexFileObject[/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class)]]
Round 2:
    input files: {}
    annotations: []
    last round: false
Processor lombok.launch.AnnotationProcessorHider$AnnotationProcessor matches [] and returns false.
[loading ZipFileIndexFileObject[/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/lib/ct.sym(META-INF/sym/rt.jar/java/lang/Object.class)]]
[loading ZipFileIndexFileObject[/root/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/lib/ct.sym(META-INF/sym/rt.jar/java/lang/String.class)]]
Round 3:

fyi @fengxue-IS

[fengxue-IS]

I have run the mvn compile command with JIT and GC disabled using -Xint -Xgcpolicy:nogc Added trace info during initialization of accessCheckCache and all instances of accessCheckCache are given a valid reference at time of creation. Using watchpoint in gdb doesn't show any modification after initial creation. Will test with the simplified test @babsingh provided

[DanHeidinga]

@fengxue-IS are you able to modify the Lombok jar? It may be sufficient to recompile lombok.launch.AnnotationProcessor.java with the lombak jar on the javac classpath.

If you can, try modifying this method to make it a no-op and confirm the issue still occurs: https://github.com/rzwitserloot/lombok/blob/20cce2049de56d5a71c6dcc376d6f4088d4552bc/src/launch/lombok/launch/AnnotationProcessor.java#L83

If it doesn't, then it would be good to log the values used in the Unsafe call to see if they make sense or if anything is wrong in the data being used.

[fengxue-IS]

I traced the source of this issue to https://github.com/rzwitserloot/lombok/blob/master/src/utils/lombok/permit/Permit.java#L68

Where call to setAccessible() run the code: UNSAFE.putBoolean(accessor, ACCESSIBLE_OVERRIDE_FIELD_OFFSET, true);

The problem with this is that the ACCESSIBLE_OVERRIDE_FIELD_OFFSET is suppose to represent the field offset of override variable, which is not accessible starting from Java 12

lombok used a hacky way which is to use Fake.class to guess the actual class shape.

static class Fake {
    boolean override;
}

// The below seems very risky, but for all AccessibleObjects in java today it does work, and starting with JDK12, making the field accessible is no longer possible.
try {
    return UNSAFE.objectFieldOffset(Fake.class.getDeclaredField("override"));
} catch (Throwable t) {
    throw saved;
}

The produced offset so happens that it maps to accessCheckCache field in OpenJ9.

[DanHeidinga]

@fengxue-IS if you update the Fake class to be:

static class Fake {
    boolean override;
    Object accessCheckCache;
}

does the problem go away? I suspect it will as it appears to be caused by the difference in how JDKs lay out fields.

If that resolves the issue, then we should get a PR opened at Lombok with this fix. There was a recent article indicating Hotspot is changing their field layout algorithms as well so having the object field may be safer even on newer Hotspots

[fengxue-IS]

@DanHeidinga I've recompiled lombok.jar with the changes you suggested and the javac test passed successfully with updated jar. I will open a PR against the lombok project with this fix.

[fengxue-IS]

Created PR against Lombok https://github.com/rzwitserloot/lombok/pull/2437

[DanHeidinga]

Thanks @fengxue-IS. And also thanks to @sudhirtumati, @GaborSzabo7, ghunteranderson & @babsingh for reporting this and helping to track it down.

I'm going to close this as this Jack has opened a PR to address this in Lombok.