search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.26k stars 717 forks source link

revisit disabled tests #10917

Closed llxia closed 1 week ago

llxia commented 3 years ago

We should revisit disabled tests to see if the issues are resolved. If they are, we need to re-enable the tests. For these ones that do not have any comments, we should check them again.

For example: https://github.com/eclipse/openj9/blob/ad7c3f09279a54f8d4d253af03da2a2a31f86f8e/test/functional/cmdLineTests/modularityddrtests/playlist.xml#L26 https://github.com/eclipse/openj9/blob/e2c38649abb06128ee30c06691b6dcc9e19db55b/test/functional/VM_Test/playlist.xml#L148 https://github.com/eclipse/openj9/blob/ab7f03ab6faa069307a3ddedf09ac26f79b9702c/test/functional/cmdLineTests/pltest/playlist.xml#L175

llxia commented 1 year ago

@KapilPowar could you take a look at this issue? Thanks

KapilPowar commented 1 year ago

@pshipton

While testing VM test suite for JDK8 we are seeing test case failures. https://github.com/eclipse/openj9/blob/e2c38649abb06128ee30c06691b6dcc9e19db55b/test/functional/VM_Test/playlist.xml#L148

Here is the grinder link

Test cases enabled -- recreateClassTest (1/20 failed ) CFdumptest (1/8 failed)

There was 1 failure:
17:13:10  1) testInnerClasses(com.ibm.j9.recreateclass.tests.RecreateClassCompareTest)junit.framework.AssertionFailedError: Incorrect number of mismatches found expected:<1> but was:<5>
17:13:10    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.compareClassFiles(RecreateClassCompareTest.java:63)
17:13:10    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runCfdumpAndCompareTools(RecreateClassCompareTest.java:74)
17:13:10    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:86)
17:13:10    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:79)
17:13:10    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.testInnerClasses(RecreateClassCompareTest.java:154)
17:13:10    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:13:10    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:13:10    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:13:10  
17:13:10  FAILURES!!!
17:13:10  Tests run: 20,  Failures: 1,  Errors: 0
--------------------------------------------------------------------
CFdumptest (1/8 failed)

There was 1 failure:
17:13:11  1) testInvalidClassFile(com.ibm.j9.cfdump.tests.lineardump.LinearDumpTests)junit.framework.AssertionFailedError: Expected: <null> but was: bad magic number
17:13:11    at com.ibm.j9.cfdump.tests.lineardump.LinearDumpTests.testInvalidClassFile(LinearDumpTests.java:123)
17:13:11    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:13:11    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:13:11    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:13:11  
17:13:11  FAILURES!!!
17:13:11  Tests run: 8,  Failures: 1,  Errors: 0

Please confirm if it is product issue.

pshipton commented 1 year ago

How do I re-run these tests? The grinder link is expired. When I try to run them every test fails with Error: cfdump not found on java.library.path!. I see that cfdump is part of the test image.

https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2649

KapilPowar commented 1 year ago

In playlist.xml file please try to add cfdump in the lib path - using $(ADD_JVM_LIB_DIR_TO_LIBPATH) \ in<command> for the test cases.

e.g. <command>$(ADD_JVM_LIB_DIR_TO_LIBPATH) \ $(JAVA_COMMAND) $(JVM_OPTIONS) \

KapilPowar commented 1 year ago

Here are latest Grinder Links aarch64_linux --Java 8 x86-64_linux -- Java 8 x86-64_windows -- Java 8

pshipton commented 1 year ago

Failure details from recreateClassTest

13:15:46  Start testInnerClasses
13:15:46  Running command: /home/jenkins/workspace/Grinder/openjdkbinary/openjdk-test-image/openj9/cfdump -t:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16917741452730/recreateClassTest_0/InnerClassesTest /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16917741452730/recreateClassTest_0/InnerClassesTest.class
13:15:46  cfdump: Wrote 1212 bytes to output file /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16917741452730/recreateClassTest_0/InnerClassesTest.j9class
13:15:46  Inner class access flags differ: 0X8, 0X0
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class access flags differ: 0X8, 0X0
13:15:46  Inner class inner name differ: LocalClass, null
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class inner name differ: LocalClass, null
13:15:46  Inner class access flags differ: 0X8, 0X0
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class access flags differ: 0X8, 0X0
13:15:46  Inner class access flags differ: 0X9, 0X0
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class access flags differ: 0X9, 0X0
13:15:46  Inner class inner name differ: InnerClass, null
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class inner name differ: InnerClass, null
13:15:46  Mismatches spotted: 5
pshipton commented 1 year ago

@TobiAjila can someone pls take a look at this recreateClassTest failure in the previous comment, it appears we may need a fix.

My changes to run the tests and add debug are in https://github.com/pshipton/openj9/tree/tests

tajila commented 1 year ago

@hangshao0 Can you please take a look at this

pshipton commented 1 year ago

Re-enabling CFdumptest via https://github.com/eclipse-openj9/openj9/pull/17939

hangshao0 commented 1 year ago

I will look at this.

hangshao0 commented 1 year ago

I've manually run cfdump on the inner class. Looking at the generated j9class and original class file:

cfdump -d 'InnerClassesTest$InnerClass.class'
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 17
Access Flags: 0x21 (
  access: public super )
ThisClass: 2 -> InnerClassesTest$InnerClass
SuperClass: 3 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (1):
  Name: 4 -> <init>
  Signature: 5 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 2

Attributes (2):
  SourceFile:
    Source File: 9 -> InnerClassesTest.java
  InnerClasses:
    Inner Class: 2 -> InnerClassesTest$InnerClass
      Outer Class: 11 -> InnerClassesTest
      Inner Name: 13 -> InnerClass
      Inner Class Access Flags: 0x9 (
  access: public )
cfdump -d 'InnerClassesTest$InnerClass.j9class'
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 17
Access Flags: 0x21 (
  access: public super )
ThisClass: 5 -> InnerClassesTest$InnerClass
SuperClass: 2 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (1):
  Name: 15 -> <init>
  Signature: 3 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 2

Attributes (2):
  InnerClasses:
    Inner Class: 5 -> InnerClassesTest$InnerClass
      Outer Class: 12 -> InnerClassesTest
      Inner Name: 8 -> InnerClass
      Inner Class Access Flags: 0x9 (
  access: public )
  SourceFile:
    Source File: 13 -> InnerClassesTest.java

I see the access flags and inner class names are the same. The differences might be from the outer class that cause the test to fail.

hangshao0 commented 1 year ago 
cfdump -d InnerClassesTest.class
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 24
Access Flags: 0x21 (
  access: public super )
ThisClass: 3 -> InnerClassesTest
SuperClass: 4 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (2):
  Name: 9 -> <init>
  Signature: 10 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 1

  Name: 13 -> getLocalClass
  Signature: 14 -> ()Ljava/lang/Class;
  Access Flags: 0x9 (
  access: public static )
  Attributes (2):
    Code:
      Max stack: 1
      Max Locals: 0
      Code Length: 3
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 8
    Signature:
      Signature: 16 -> ()Ljava/lang/Class<*>;

Attributes (2):
  SourceFile:
    Source File: 18 -> InnerClassesTest.java
  InnerClasses:
    Inner Class: 5 -> InnerClassesTest$InnerClass
      Outer Class: 3 -> InnerClassesTest
      Inner Name: 6 -> InnerClass
      Inner Class Access Flags: 0x9 (
  access: public )
    Inner Class: 2 -> InnerClassesTest$1LocalClass
      Outer Class: (not a member)
      Inner Name: 8 -> LocalClass
      Inner Class Access Flags: 0x0 (
  access: default )
cfdump -d InnerClassesTest.j9class
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 22
Access Flags: 0x21 (
  access: public super )
ThisClass: 7 -> InnerClassesTest
SuperClass: 3 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (2):
  Name: 21 -> <init>
  Signature: 14 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 1

  Name: 12 -> getLocalClass
  Signature: 13 -> ()Ljava/lang/Class;
  Access Flags: 0x9 (
  access: public static )
  Attributes (2):
    Code:
      Max stack: 1
      Max Locals: 0
      Code Length: 3
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 8
    Signature:
      Signature: 4 -> ()Ljava/lang/Class<*>;

Attributes (2):
  InnerClasses:
    Inner Class: 20 -> InnerClassesTest$InnerClass
      Outer Class: 7 -> InnerClassesTest
      Inner Name: (anonymous)                 <------------------------------------- this is different from the original class
      Inner Class Access Flags: 0x0 (         <------------------------------------- this is different from the original class
  access: default )
    Inner Class: 1 -> InnerClassesTest$1LocalClass
      Outer Class: (not a member)
      Inner Name: (anonymous)
      Inner Class Access Flags: 0x0 (
  access: default )
  SourceFile:
    Source File: 5 -> InnerClassesTest.java

The difference is from the inner class attribute of the outer class. The inner class simple name and access flags are not preserved in the romClass of the outer class: https://github.com/eclipse-openj9/openj9/blob/949a4df06ef763ed4f3a83faf8d9534f911f4d07/runtime/bcutil/ClassFileWriter.cpp#L1054-L1056

If we decided to not record these information in the romClass in #13199, the test needs to be updated to accept such differences in .class and .j9class files.

hangshao0 commented 1 year ago

@ChengJin01 ^^

llxia commented 12 months ago

CFdumptest is only enabled for JDK8. I think it is able to run against all versions. https://github.com/pshipton/openj9/blob/master/test/functional/VM_Test/playlist.xml#L167 @KapilPowar could you run CFdumptest on JDK11 and JDK17 when you have time? Thanks

pshipton commented 12 months ago

I tried other versions but there are problems that would need to be resolved.

KapilPowar commented 11 months ago

Grinder test runs for CFdumptest JDK11 x86-64_linux JDK11 x86-64_mac JDK11 x86-64_windows JDK17 x86-64_linux JDK17 x86-64_mac JDK17 x86-64_windows

llxia commented 11 months ago

Thanks @KapilPowar . Could also run on other missing platforms in Grinder? Thanks

KapilPowar commented 11 months ago

Grinder test runs for CFdumptest JDK17 s390x_linux --UNSTABLE aarch64_mac --Success ppc64_aix --UNSTABLE

JDK11 aarch64_mac -- Success s390x_linux--UNSTABLE ppc64_aix --UNSTABLE

KapilPowar commented 11 months ago

@llxia @pshipton FYI Grinder test run for recreateClassTest still results in same error -- JDK8 x86-64_linux


There was 1 failure:
17:01:36  1) testInnerClasses(com.ibm.j9.recreateclass.tests.RecreateClassCompareTest)junit.framework.AssertionFailedError: Incorrect number of mismatches found expected:<1> but was:<5>
17:01:36    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.compareClassFiles(RecreateClassCompareTest.java:63)
17:01:36    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runCfdumpAndCompareTools(RecreateClassCompareTest.java:74)
17:01:36    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:86)
17:01:36    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:79)
17:01:36    at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.testInnerClasses(RecreateClassCompareTest.java:154)
17:01:36    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:01:36    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:01:36    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:01:36  
17:01:36  FAILURES!!!
17:01:36  Tests run: 20,  Failures: 1,  Errors: 0
17:01:36  
17:01:36  -----------------------------------
17:01:36  recreateClassTest_0_FAILED
17:01:36  -----------------------------------
pshipton commented 11 months ago

I assume this can be moved forward.

pshipton commented 11 months ago

Re https://github.com/eclipse-openj9/openj9/issues/10917#issuecomment-1725335609, we're waiting for @ChengJin01 to have time to take a look, based on https://github.com/eclipse-openj9/openj9/issues/10917#issuecomment-1681233241

ChengJin01 commented 11 months ago

I agree with @hangshao0 's investigation & conclusions given we don' record the access flags in the romClass by default.

pshipton commented 11 months ago

So the next step is to update the test itself to ignore those differences. @TobiAjila

KapilPowar commented 11 months ago

@llxia Grinder test run for cmdLineTester_forceLazySymbolResolution JDK17 x86-64_linux

Grinder test run for cmdLineTester_jvmtitests_hcr_OSRG_nongold JDK17 x86-64_linux JDK17 x86-64_mac JDK17 x86-64_windows

llxia commented 11 months ago

@KapilPowar if the Grinders pass, please create PR to reenable the tests. Thanks

KapilPowar commented 11 months ago

@KapilPowar if the Grinders pass, please create PR to reenable the tests. Thanks

PR created https://github.com/eclipse-openj9/openj9/pull/18181

KapilPowar commented 10 months ago

@llxia Grinder test run for TestFlushReflectionCache & jit_jitt_openj9_none_SCC x86-64_linux x86-64_mac x86-64_windows

PR created #18206

KapilPowar commented 10 months ago

@llxia Grinder test run for cmdLineTester_dumpromclasstests aix_ppc-64 JDK8 aix_ppc-64 JDK11 aix_ppc-64 JDK17

PR created #18215

tajila commented 3 months ago

@llxia Do we still need this issue?

llxia commented 1 week ago

I believe this issue can be closed.