search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

openJcePlusTests_0_PASSED but there were a number of junit failures #19596

Open JasonFengJ9 opened 5 months ago

JasonFengJ9 commented 5 months ago

Failure link

From an internal build(sles15le-svl-rt4-1):

02:29:14  java version "21.0.4-beta" 2024-07-16
02:29:14  IBM Semeru Runtime Certified Edition 21.0.4+5-202405310031 (build 21.0.4-beta+5-202405310031)
02:29:14  Eclipse OpenJ9 VM 21.0.4+5-202405310031 (build master-ff7726e0f2, JRE 21 AIX ppc64-64-Bit Compressed References 20240531_184 (JIT enabled, AOT enabled)
02:29:14  OpenJ9   - ff7726e0f2
02:29:14  OMR      - 6463b7635
02:29:14  JCL      - 2ac55d2d7 based on jdk-21.0.4+5)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2024-05-31T07:02:18.469Z] variation: NoOptions
[2024-05-31T07:02:18.469Z] JVM_OPTIONS:  

[2024-05-31T07:03:13.596Z]      [test]     [junit] Running gcm_oneReadByteCorrupt test
[2024-05-31T07:03:13.596Z]      [test]     [junit]   Fail. No exception thrown.

...

[2024-05-31T07:03:13.597Z]      [test]     [junit] Running cbc_readAllIllegalBlockSize test
[2024-05-31T07:03:13.597Z]      [test]     [junit]   Fail. No exception thrown.

[2024-05-31T07:05:58.857Z]      [test]     [junit] Running gcm_AEADBadTag
[2024-05-31T07:05:58.857Z]      [test]     [junit]   Fail. No exception thrown.

[2024-05-31T07:43:31.951Z]      [test]     [junit] Tests run: 1705, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,692.66 sec
[2024-05-31T07:43:31.951Z]      [test]      [echo] TestAll COMPLETED
[2024-05-31T07:43:31.951Z]      [test]      [echo] ALL TESTS COMPLETED
[2024-05-31T07:43:31.951Z]      [test] 
[2024-05-31T07:43:31.951Z]      [test] BUILD SUCCESSFUL
[2024-05-31T07:43:31.951Z]      [test] Total time: 41 minutes 1 second
[2024-05-31T07:43:31.951Z] 
[2024-05-31T07:43:31.951Z] BUILD SUCCESSFUL
[2024-05-31T07:43:31.951Z] Total time: 41 minutes 3 seconds
[2024-05-31T07:43:31.951Z] -----------------------------------
[2024-05-31T07:43:31.951Z] openJcePlusTests_0_PASSED

FYI @jasonkatonica @KostasTsiounis

KostasTsiounis commented 5 months ago

Those are some tests that have some problems and need to be fixed. We have identified them and opened an issue for them. There is no need to worry at the time, as they are not testing what is expected and the printed failures are not actually failures.

@jasonkatonica Do you agree?

jasonkatonica commented 5 months ago

Hello,

For these particular tests listed above I agree @KostasTsiounis . These messages in my view are incorrect and the test is not really failing. We do have an issue opened for the failure and these tests should be updated accordingly in the near future.

JasonFengJ9 commented 5 months ago

Thanks for the clarifications.

On a separate question for triage purposes, what kind of output message indicates a failure and an issue should be opened?

KostasTsiounis commented 5 months ago

In this line 

[junit] Tests run: 1705, Failures: 0, Errors: 0, Skipped: 0

the failures or errors won't be zero. I think everything else is tests that are a bit off and will require fixing on our end.

JasonFengJ9 commented 5 months ago

JDK22 ppc64_aix

[2024-06-02T04:43:57.911Z]      [test]     [junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,883.586 sec

It isn't clear what failure it is, any hint?

jasonkatonica commented 5 months ago

It seems like the ant junit runner is different then the default maven one we typically use. Typically we see errors printed in the logs when a failure occurs in a stack trace form. We do see this exact testRSAPSSInterop2 test fail on occasion. There seems to be hang encountered on occasion with the RSAPSS suite. I would assume that we are enountering this known issue although without stderr being printed its hard to know for sure:

https://github.ibm.com/runtimes/jit-crypto/issues/431

This being said we should investigate why the tooling from ant is not printing to stderr it appears.

jasonkatonica commented 5 months ago

One other thing we have noticed recently is that once a test fails the tests do not continue on to run the other variations. We plan on investigating that also as we want all the tests to run even if a test fails.

JinhangZhang commented 4 months ago

As @jasonkatonica mentioned, we made a change to fix this unexpected behaviour.

One other thing we have noticed recently is that once a test fails the tests do not continue on to run the other variations. We plan on investigating that also as we want all the tests to run even if a test fails.

We made a grinder test run in OpenJcePlusTest_0 and force some of the tests to be failed in there. In this case, we can monitor if the unexpected behavior has been addressed or not. As you may notice in https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_FIPS/1656/. The grinder jobs TEST TARGETS SUMMARYstates a pass. But there are 5 failures listed at the bottom.

Not sure how this contradiction will be treated? Will this be treated as a failure or not?