Open cjjdespres opened 3 months ago
A similar NoSuchMethodError: java/nio/ByteBuffer.rewind()Ljava/nio/ByteBuffer
openjdk8_j9_special.system_x86-64_linux
[2024-07-18T01:01:04.860Z] variation: Mode301
[2024-07-18T01:01:04.861Z] JVM_OPTIONS: -Xgcpolicy:metronome -Xnocompressedrefs
[2024-07-18T01:01:12.939Z] LT 18:01:12.213 - First failure detected by thread: load-2. Not creating dumps as no dump generation is requested for this load test
[2024-07-18T01:01:12.939Z] LT 18:01:12.214 - Test failed
[2024-07-18T01:01:12.939Z] LT Failure num. = 1
[2024-07-18T01:01:12.939Z] LT Test number = 3053
[2024-07-18T01:01:12.939Z] LT Test details = 'Mauve[gnu.testlet.java.nio.ByteBuffer.Order]'
[2024-07-18T01:01:12.939Z] LT Suite number = 0
[2024-07-18T01:01:12.939Z] LT Thread number = 2
[2024-07-18T01:01:12.939Z] LT >>> Captured test output >>>
[2024-07-18T01:01:12.939Z] LT Test failed:
[2024-07-18T01:01:12.939Z] LT java.lang.NoSuchMethodError: java/nio/ByteBuffer.rewind()Ljava/nio/ByteBuffer; (loaded from /home/jenkins/workspace/Test_openjdk8_j9_special.system_x86-64_linux_testList_9/jdkbinary/j2sdk-image/jre/lib/rt.jar by <Bootstrap Loader>) called from class gnu.testlet.java.nio.ByteBuffer.Order (loaded from file:/home/jenkins/testDependency/system_lib/mauve/mauve.jar by sun.misc.Launcher$AppClassLoader@e211a34).
[2024-07-18T01:01:12.939Z] LT at gnu.testlet.java.nio.ByteBuffer.Order.test(Order.java:40)
[2024-07-18T01:01:12.939Z] LT at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2024-07-18T01:01:12.939Z] LT at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2024-07-18T01:01:12.939Z] LT at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-07-18T01:01:12.939Z] LT at java.lang.reflect.Method.invoke(Method.java:503)
[2024-07-18T01:01:12.939Z] LT at net.adoptopenjdk.loadTest.adaptors.MauveAdaptor.executeTest(MauveAdaptor.java:74)
[2024-07-18T01:01:12.939Z] LT at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:182)
[2024-07-18T01:01:12.939Z] LT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2024-07-18T01:01:12.939Z] LT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2024-07-18T01:01:12.939Z] LT at java.lang.Thread.run(Thread.java:826)
[2024-07-18T01:01:12.939Z] LT <<<
[2024-07-18T01:01:16.290Z] MauveMultiThrdLoad_special_5m_9_FAILED
openjdk8_j9_special.system_x86-64_linux
[2024-07-18T01:01:40.496Z] LT java.lang.NoSuchMethodError: java/nio/ByteBuffer.position(I)Ljava/nio/ByteBuffer; (loaded from /home/jenkins/workspace/Test_openjdk8_j9_special.system_x86-64_linux_testList_9/jdkbinary/j2sdk-image/jre/lib/rt.jar by <Bootstrap Loader>) called from class gnu.testlet.java.nio.ByteBuffer.Allocating (loaded from file:/home/jenkins/testDependency/system_lib/mauve/mauve.jar by sun.misc.Launcher$AppClassLoader@de843f5).
Given the intermittent nature, I suspect JIT but not sure. Or maybe a machine/file system/network problem. @hzongaro fyi, the failure rate is high, could this be JIT?
The methods are inherited from java.nio.Buffer
I think the rewind
and position
methods inherited from java.nio.Buffer
would have Ljava/nio/Buffer;
as their return types. I'm not sure where those signatures might have come from. . . . Taking a look.
There's something fishy here. I was able to produce a core file with a grinder run specifying
-Xdump:jit+java+system:events=throw+systhrow,filter=java/lang/NoSuchMethodError#gnu/testlet/java/nio/ByteBuffer*
in EXTRA_OPTIONS
.
Using jdmpview
to look at a core file that was produced by that grinder run, I saw the following call stack on one of the threads:
> <3ffb4a6e800> Generic special frame
<3ffb4a6e800> !j9method 0x000003FE6441C710 gnu/testlet/java/nio/ByteBuffer/compact.test(Lgnu/testlet/TestHarness;)V
<3ffb4a6e800> JNI call-in frame
<3ffb4a6e800> !j9method 0x000003FFB42E7FA0 sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<3ffb4a6e800> !j9method 0x000003FFB42E7F60 sun/reflect/NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<3ffb4a6e800> !j9method 0x000003FFB42D7248 sun/reflect/DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<3ffb4a6e800> !j9method 0x000003FFB4232400 java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
<3ffb4a6e800> !j9method 0x000003FFB4986818 net/adoptopenjdk/loadTest/adaptors/MauveAdaptor.executeTest()Lnet/adoptopenjdk/loadTest/adaptors/AdaptorInterface$ResultStatus;
<3ffb4a6e800> !j9method 0x000003FFB4A37618 net/adoptopenjdk/loadTest/LoadTestRunner$2.run()V
<3ffb4a6e800> !j9method 0x000003FFB48D5AD8 java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
<3ffb4a6e800> !j9method 0x000003FFB4A34800 java/util/concurrent/ThreadPoolExecutor$Worker.run()V
<3ffb4a6e800> !j9method 0x000003FFB421B970 java/lang/Thread.run()V
Looking at the j9method and bytecodes for ByteBuffer/compact.test
:
> !j9method 0x000003FE6441C710
J9Method at 0x3fe6441c710 {
Fields for J9Method:
0x0: U8* bytecodes = !j9x 0x000003FE64410EAC // "
�"
0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x000003FE78240AC0 (flags = 0x0)
0x10: void* methodRunAddress = !j9x 0x0000000000000006
0x18: volatile void* extra = !j9x 0x00000000000007CF
}
Signature: gnu/testlet/java/nio/ByteBuffer/compact.test(Lgnu/testlet/TestHarness;)V !bytecodes 0x000003FE6441C710
ROM Method: !j9rommethod 0x000003FE64410E98
Next Method: !j9method 0x000003FE6441C730
> !bytecodes 0x000003FE6441C710
Name: test
Signature: (Lgnu/testlet/TestHarness;)V
Access Flags (1050001): public
Internal Attribute Flags:
Max Stack: 4
Argument Count: 2
Temp Count: 1
0 bipush 10
2 invokestatic 5 java/nio/ByteBuffer.allocate(I)Ljava/nio/ByteBuffer;
5 astore2
6 aload2
7 iconst0
8 invokevirtual 7 java/nio/ByteBuffer.position(I)Ljava/nio/ByteBuffer; <<<<
11 pop
12 aload2
13 iconst3
...
Note the signature of the call to ByteBuffer.position(I)
is java/nio/ByteBuffer.position(I)Ljava/nio/ByteBuffer;
. But that method is inherited from java/nio/Buffer
in JDK 8 and 11, where it has a return type of java/nio/Buffer
.
I tried building and running the test locally, I examined the class file for gnu/testlet/java/nio/ByteBuffer/compact.test
in mauve.jar
with javap
, and saw the expected signature on the call to position(I)
:
public void test(gnu.testlet.TestHarness);
Code:
0: bipush 10
2: invokestatic #2 // Method java/nio/ByteBuffer.allocate:(I)Ljava/nio/ByteBuffer;
5: astore_2
6: aload_2
7: iconst_0
8: invokevirtual #3 // Method java/nio/ByteBuffer.position:(I)Ljava/nio/Buffer; <<<<
11: pop
12: aload_2
13: iconst_3
I tried another internal grinder run, requesting that the workspace be preserved. I signed into the machine, and examined the class file for gnu/testlet/java/nio/ByteBuffer/compact.test
in mauve.jar
with javap
, and saw the wrong signature for ByteBuffer.position(I)
:
public void test(gnu.testlet.TestHarness);
descriptor: (Lgnu/testlet/TestHarness;)V
flags: ACC_PUBLIC
Code:
stack=4, locals=3, args_size=2
0: bipush 10
2: invokestatic #7 // Method java/nio/ByteBuffer.allocate:(I)Ljava/nio/ByteBuffer;
5: astore_2
6: aload_2
7: iconst_0
8: invokevirtual #13 // Method java/nio/ByteBuffer.position:(I)Ljava/nio/ByteBuffer;
11: pop
12: aload_2
13: iconst_3
Looking at the console log to understand where mauve.jar
came from, I see this:
09:24:12 --------------------------------------------
09:24:12 Starting download third party dependent jars
09:24:12 --------------------------------------------
09:24:12 downloading dependent third party jars to /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/json-simple.jar exists with correct hash, not downloading
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/cvsclient/org-netbeans-lib-cvsclient.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/apache-ant/lib/ant-launcher.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/log4j/log4j-core.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/junit/junit.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/asm/asm.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/jcstress-tests-all-20240222.jar exists with correct hash, not downloading
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/mauve/mauve.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/junit/hamcrest-core.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/log4j/log4j-api.jar exists, not downloading.
09:24:12 /home/jenkins/workspace/Grinder_iteration_2/../../testDependency/system_lib/tools/tools.jar exists, not downloading.
Looking at the date of the file, I see this:
jenkins@rtj-ubu24s390x-svl-test-6lslg-1:~/workspace/Grinder_iteration_2$ ls -lta /home/jenkins/testDependency/system_lib/mauve/mauve.jar
-rw-rw-r-- 1 jenkins jenkins 17477855 Jun 26 18:07 /home/jenkins/testDependency/system_lib/mauve/mauve.jar
I'm wondering where the mauve.jar
came from. Is it possible it was built with javac
from JDK 17 or later, with the --release 8
option to allow it to be used with older versions?
For now, I don't think this is a JIT problem.
@llxia can someone take a look pls.
A similar NoSuchMethodError
JDK8 aarch64_linux(rhel9-aarch64-1
)
[2024-08-03T06:01:46.507Z] variation: Mode101
[2024-08-03T06:01:46.507Z] JVM_OPTIONS: -Xjit -Xgcpolicy:optthruput -Xnocompressedrefs
[2024-08-03T06:01:55.825Z] LT 02:01:55.609 - First failure detected by thread: load-4. Not creating dumps as no dump generation is requested for this load test
[2024-08-03T06:01:55.825Z] LT 02:01:55.610 - Test failed
[2024-08-03T06:01:55.825Z] LT Failure num. = 1
[2024-08-03T06:01:55.825Z] LT Test number = 3068
[2024-08-03T06:01:55.825Z] LT Test details = 'Mauve[gnu.testlet.java.nio.LongBuffer.compact]'
[2024-08-03T06:01:55.825Z] LT Suite number = 0
[2024-08-03T06:01:55.825Z] LT Thread number = 4
[2024-08-03T06:01:55.825Z] LT >>> Captured test output >>>
[2024-08-03T06:01:55.825Z] LT Test failed:
[2024-08-03T06:01:55.825Z] LT java.lang.NoSuchMethodError: java/nio/LongBuffer.position(I)Ljava/nio/LongBuffer; (loaded from /home/jenkins/workspace/Test_openjdk8_j9_special.system_aarch64_linux_testList_9/jdkbinary/j2sdk-image/jre/lib/rt.jar by <Bootstrap Loader>) called from class gnu.testlet.java.nio.LongBuffer.compact (loaded from file:/home/jenkins/testDependency/system_lib/mauve/mauve.jar by sun.misc.Launcher$AppClassLoader@4fe4b29c).
[2024-08-03T06:01:55.825Z] LT at gnu.testlet.java.nio.LongBuffer.compact.test(compact.java:35)
[2024-08-03T06:01:55.825Z] LT at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2024-08-03T06:01:55.825Z] LT at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2024-08-03T06:01:55.825Z] LT at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-08-03T06:01:55.825Z] LT at java.lang.reflect.Method.invoke(Method.java:503)
[2024-08-03T06:01:55.825Z] LT at net.adoptopenjdk.loadTest.adaptors.MauveAdaptor.executeTest(MauveAdaptor.java:74)
[2024-08-03T06:01:55.825Z] LT at net.adoptopenjdk.loadTest.LoadTestRunner$2.run(LoadTestRunner.java:182)
[2024-08-03T06:01:55.825Z] LT at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2024-08-03T06:01:55.825Z] LT at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2024-08-03T06:01:55.825Z] LT at java.lang.Thread.run(Thread.java:826)
[2024-08-03T06:01:55.825Z] LT <<<
[2024-08-03T06:01:58.120Z] MauveMultiThrdLoad_special_5m_0_FAILED
This kind of errors out there are normally when a code is compiled with higher version then 8 with -source 8 and -target 8. Once one compiles it with --release 8, the signatures should be correct. Since Java 9, the position(integer), flip, rewind, ... methods have overrides returning ByteBuffer in the ByteBuffer class. Until 8, they are inherited from Buffer and thus are returning Buffer. It means that SOMETHING in the chain is built with java >= 9 and not using --release 8. If the --release 8 is not possible, the code itself can cast the ByteBuffer to Buffer in the places those methods are called and it will work with java 8 after.
@llxia reminder
Sorry, this slipped off my radar.
mauve.jar
got built in https://openj9-jenkins.osuosl.org/job/systemtest.getDependency/45/
It looks like Temurin JDK8 is used.
+ /home/jenkins/workspace/systemtest.getDependency/j2sdk-image/jre/bin/java -version
openjdk version "1.8.0_422"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_422-b05)
OpenJDK 64-Bit Server VM (Temurin)(build 25.422-b05, mixed mode)
@annaibm could you confirm JDK8 was used for building mauve.jar
? https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/getDependency#L20.
@annaibm confirmed that JDK8 is used to compile mauve (we did not use a higher version of Java to compile)
build-tests:
[exec] openjdk version "1.8.0_422"
[exec] OpenJDK Runtime Environment (Temurin)(build 1.8.0_422-b05)
[exec] OpenJDK 64-Bit Server VM (Temurin)(build 25.422-b05, mixed mode)
[javac] Compiling 6694 source files to /tmp/mauve/mauve
Currently, we are trying to use JDK17 with --release 8 option to see if it works. If you have a better suggestion, please let us know.
Ok, so the analysis of this problem is following: It is the class containing gnu.testlet.java.nio.LongBuffer.compact.test(compact.java:35) that was built with java > 1.8 and maybe with source 8/target 8. In Java 8, the java.nio.LongBuffer.position(int) was inherited from java.nio.Buffer. So the return type was java.nio.Buffer. Jdk 9 introduced overridden methods with covariant return types for some of the functions that include the position(int). If you compile with source/target 8, you will get the right bytecode version, right language features, but you might get the APIs wrong. That is why --release was introduced. The error message says that the code that calls that position(int) expects the function to return java.nio.LongBuffer, which will not happen if it is run with Java 8. So, the fix is to compile the code that calls that function either with JDK8, or with JDK9+ with --release 8. Like that the compiler will resolve the call to that function as being inherited from java.nio.Buffer (and having the java.nio.Buffer return type) The classes that got in JDK9 the overriden methods are java.nio.ByteBuffer, java.nio.CharBuffer and java.nio.LongBuffer. Possible others, but I did not hit the problem on them for the while. I hope this explanation helps with understanding where the actual problem lies. It is not the bytecode version, it is the APIs that are not compatible with JDK8
So, that pull request basically fixes the build of the mauve.jar, so that it works with the java 8. It is hanging in eclipsefdn/eca. I am not sure whether we (SUSE) have ever signed something like this. I also don't care to be credited, so anybody who has the right paperwork on file can submit that patch. I transfer automatically the copyright to anybody brave enough to get it integrated.
Now for the record: this is one of the most annoying improvements of Java 9. The release=8 should solve it, but it hides from the user many of apis that were considered internal or deprecated, so it is not always possible to use it. When it is only one file that is problematic and one is lucky not to have any cross-dependencies between this file and the rest, one can do what I did. But this is not always the case. In that situation, there is still a fix that works with Java 8 and Java 9+, but it requires source changes. At any invocation of one of the following methods: position(int newPosition) limit(int newLimit) flip() clear() mark() reset() rewind(), the object must be cast to Buffer and thus the right method will be called. It is cumbersome but in some situations, the only solution.
The failing tests:
MiniMix_5m_0
andMiniMix_10m_1
inopenjdk8_j9_extended.system_s390x_linux_jit
MauveSingleThrdLoad_J9_5m_0
,MauveSingleInvocLoad_J9_5m_1
, andMauveMultiThrdLoad_5m_0
, inopenjdk8_j9_sanity.system_s390x_linux_jit
These are in the JITServer nightly tests, but a 100x grinder of
MiniMix_5m_0,MiniMix_5m_1
without JITServer still produced failures (20/200).The earliest failures I can find in the JITServer nightly tests in the two suites are:
https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk8_j9_extended.system_s390x_linux_jit_Personal/1062/ https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk8_j9_sanity.system_s390x_linux_jit_Personal/1065/
The commit ranges between the earliest failures and the previous successes are 69cba86c0f1...d946965c3ae and https://github.com/eclipse/omr/compare/770936378ec...38148eaaf3b.
Console log for
MauveSingleThrdLoad_J9_5m_0
in the earlysanity.system
failure above: