search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

Core file issue on Mac - error getting thread command data #13811

Open pshipton opened 3 years ago

pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/126 - osx1013-x86-3 testSCCMLTests4_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/126/functional_test_output.tar.gz

Testing: Test 180-e: Create a system dump by using a shared cache with enableBCI option and without a JVMTI agent
Test start time: 2021/10/28 06:16:09 Eastern Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs  -Xmx4m -Xdump:system:file=shrcmltest.dmp,events=vmstop -Xshareclasses:name=ShareClassesCMLTests,reset,enableBCI -cp /Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/utils/utils.jar org.openj9.test.ivj.Hanoi 2
Time spent starting: 5 milliseconds
Time spent executing: 846 milliseconds
Test result: FAILED
 [OUT] Beginning puzzle.  Solving for 2 disks.
 [OUT] Moved disk 0 to 1
 [OUT] Moved disk 0 to 2
 [OUT] Moved disk 1 to 2
 [OUT] Puzzle solved!
 [ERR] JVMSHRC023E Cache does not exist
 [ERR] JVMDUMP039I Processing dump event "vmstop", detail "#0000000000000000" at 2021/10/28 06:16:10 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/output_16354069339138/testSCCMLTests4_0/shrcmltest.dmp' in response to an event
 [ERR] error getting thread command data:
 [ERR]  (os/kern) invalid argument
 [ERR] JVMDUMP012E Error in System dump: /Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/output_16354069339138/testSCCMLTests4_0/shrcmltest.dmp
 [ERR] JVMDUMP013I Processed dump event "vmstop", detail "#0000000000000000".
>> Success condition was found: [Output match: Moved disk 0 to 1]
>> Required condition was not found: [Output match: System dump written]
>> Failure condition was not found: [Output match: 0001.dmp]
>> Failure condition was not found: [Output match: corrupt]
>> Failure condition was not found: [Output match: Unhandled Exception]

@mikezhang1234567890

mikezhang1234567890 commented 3 years ago

Looks intermittent as https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_0/127/ passed on the same machine. Have also not been able to reproduce the error locally.

Failure occurs in https://github.com/eclipse/omr/blob/master/port/osx/omrosdump.c#L252 or the 2 calls below, either because the thread returned by task_threads is invalid or the thread flavour doesn't exist on the machine.

pshipton commented 3 years ago

Right, it's intermittent. We've been running the new core file creation code for a while, and haven't seen this until now. Also that isn't the only core created by the same test run.

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/99 - osx1013-x86-4

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/99/functional_test_output.tar.gz

cmdLineTester_GCCheck_3

Testing: Create core file
Test start time: 2021/12/09 19:06:55 Pacific Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:balanced -Xnocompressedrefs  -Xmx256m -cp /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdline_options_testresources/cmdlinetestresources.jar:/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/gcCheck/gcCheck.jar -Xdump:system:file=j9core.dmp,events=vmstop,request=exclusive+compact org.openj9.test.nogc.Main
Time spent starting: 5 milliseconds
***[TEST INFO 2021/12/09 19:16:55] ProcessKiller detected a timeout after 600000 milliseconds!***
INFO: The current OS is 'Mac OS X'. 'Debug on timeout' is currently only supported on Linux.
***[TEST INFO 2021/12/09 19:16:55] executing kill -ABRT 22819***
***[TEST INFO 2021/12/09 19:16:55] kill -ABRT signal sent***
Time spent executing: 714753 milliseconds
Test result: FAILED
Output from test:
 [OUT] GCCheck Main class executed successfully
 [ERR] JVMDUMP039I Processing dump event "vmstop", detail "#0000000000000000" at 2021/12/09 19:06:59 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16391054065801/cmdLineTester_GCCheck_3/j9core.dmp' in response to an event
 [ERR] JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16391054065801/cmdLineTester_GCCheck_3/j9core.dmp
 [ERR] JVMDUMP013I Processed dump event "vmstop", detail "#0000000000000000".
 [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2021/12/09 19:18:49 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16391054065801/cmdLineTester_GCCheck_3/core.20211209.191849.22819.0002.dmp' in response to an event
 [ERR] error getting thread command data:
 [ERR]  (ipc/send) invalid destination port
 [ERR] JVMDUMP012E Error in System dump: /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16391054065801/cmdLineTester_GCCheck_3/core.20211209.191849.22819.0002.dmp

cmdLineTester_GCCheck_4

Testing: Create core file
Test start time: 2021/12/09 19:19:11 Pacific Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/java"  -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced  -Xmx256m -cp /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdline_options_testresources/cmdlinetestresources.jar:/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/gcCheck/gcCheck.jar -Xdump:system:file=j9core.dmp,events=vmstop,request=exclusive+compact org.openj9.test.nogc.Main
Time spent starting: 5 milliseconds
***[TEST INFO 2021/12/09 19:29:11] ProcessKiller detected a timeout after 600000 milliseconds!***
INFO: The current OS is 'Mac OS X'. 'Debug on timeout' is currently only supported on Linux.
***[TEST INFO 2021/12/09 19:29:11] executing kill -ABRT 23354***
***[TEST INFO 2021/12/09 19:29:11] kill -ABRT signal sent***
Time spent executing: 682915 milliseconds
Test result: FAILED
Output from test:
 [OUT] GCCheck Main class executed successfully
 [ERR] JVMDUMP039I Processing dump event "vmstop", detail "#0000000000000000" at 2021/12/09 19:19:15 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16391054065801/cmdLineTester_GCCheck_4/j9core.dmp' in response to an event
 [ERR] JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16391054065801/cmdLineTester_GCCheck_4/j9core.dmp
 [ERR] JVMDUMP013I Processed dump event "vmstop", detail "#0000000000000000".
 [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2021/12/09 19:30:34 - please wait.

The second one is like https://github.com/eclipse-openj9/openj9/issues/13921

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/110/ cmdLineTester_GCCheck_3 -Xjit -Xgcpolicy:balanced -Xnocompressedrefs

Testing: Create core file
Test start time: 2021/12/21 20:27:58 Pacific Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:balanced -Xnocompressedrefs  -Xmx256m -cp /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdline_options_testresources/cmdlinetestresources.jar:/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/gcCheck/gcCheck.jar -Xdump:system:file=j9core.dmp,events=vmstop,request=exclusive+compact org.openj9.test.nogc.Main
Time spent starting: 4 milliseconds
***[TEST INFO 2021/12/21 20:37:59] ProcessKiller detected a timeout after 600000 milliseconds!***
INFO: The current OS is 'Mac OS X'. 'Debug on timeout' is currently only supported on Linux.
***[TEST INFO 2021/12/21 20:37:59] executing kill -ABRT 38969***
***[TEST INFO 2021/12/21 20:37:59] kill -ABRT signal sent***
Time spent executing: 685223 milliseconds
Test result: FAILED
Output from test:
 [OUT] GCCheck Main class executed successfully
 [ERR] JVMDUMP039I Processing dump event "vmstop", detail "#0000000000000000" at 2021/12/21 20:28:03 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/output_16401471902926/cmdLineTester_GCCheck_3/j9core.dmp' in response to an event
 [ERR] JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/output_16401471902926/cmdLineTester_GCCheck_3/j9core.dmp
 [ERR] JVMDUMP013I Processed dump event "vmstop", detail "#0000000000000000".
 [ERR] JVMDUMP039I Processing dump event "abort", detail "" at 2021/12/21 20:39:24 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_sanity.functional_x86-64_mac_Nightly_testList_0/aqa-tests/TKG/output_16401471902926/cmdLineTester_GCCheck_3/core.20211221.203924.38969.0002.dmp' in response to an event
 [ERR] error getting segment command data:
 [ERR]  (os/kern) invalid argument
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_x86-64_mac_Nightly_testList_1/307 - mac10-x86-4 testSCCMLTests4_1

Testing: Test 180-e: Create a system dump by using a shared cache with enableBCI option and without a JVMTI agent
Test start time: 2022/09/30 18:22:41 Pacific Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/java"  -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xmx4m -Xdump:system:file=shrcmltest.dmp,events=vmstop -Xshareclasses:name=ShareClassesCMLTests,reset,enableBCI -cp /Users/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/utils/utils.jar org.openj9.test.ivj.Hanoi 2
Time spent starting: 8 milliseconds
Time spent executing: 611 milliseconds
Test result: FAILED
 [OUT] Beginning puzzle.  Solving for 2 disks.
 [OUT] Moved disk 0 to 1
 [OUT] Moved disk 0 to 2
 [OUT] Moved disk 1 to 2
 [OUT] Puzzle solved!
 [ERR] JVMSHRC023E Cache does not exist
 [ERR] JVMDUMP039I Processing dump event "vmstop", detail "#0000000000000000" at 2022/09/30 18:22:42 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16645856918319/testSCCMLTests4_1/shrcmltest.dmp' in response to an event
 [ERR] error getting thread command data:
 [ERR]  (ipc/send) invalid destination port
 [ERR] JVMDUMP012E Error in System dump: /Users/jenkins/workspace/Test_openjdk17_j9_extended.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16645856918319/testSCCMLTests4_1/shrcmltest.dmp
pshipton commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_1/388 cmdLineTester_modularityddrtests11_0

Testing: Create core file --illegal-access=permit
Test start time: 2022/11/23 03:03:43 Eastern Standard Time
Running command: "/Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/bin/java"  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs  -Xmx4m -Xdump:system:file=j9corep.dmp,events=vmstop --illegal-access=permit -version
Test result: FAILED
...
 [ERR] JVMDUMP039I Processing dump event "vmstop", detail "#0000000000000000" at 2022/11/23 03:03:43 - please wait.
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16691878533665/cmdLineTester_modularityddrtests11_0/j9corep.dmp' in response to an event
 [ERR] error getting thread command data:
 [ERR]  (ipc/send) invalid destination port
 [ERR] JVMDUMP012E Error in System dump: /Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly_testList_1/aqa-tests/TKG/output_16691878533665/cmdLineTester_modularityddrtests11_0/j9corep.dmp