search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 720 forks source link

SharedClassesAPI_1 win2012-x86-8 There is not enough space on the disk #13803

Open pshipton opened 3 years ago

pshipton commented 3 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/131/ - win2012-x86-6 SharedClassesAPI_1 -Xcompressedrefs -Xjit -Xgcpolicy:gencon

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/131/system_test_output.tar.gz

STF 01:37:33.046 - +------ Step 69 - DefaultLocUtilitiesJVMTI: Monitor workload processes
STF 01:37:33.046 - | Wait for processes to meet expectations
STF 01:37:33.046 - |   Processes: [WL1, WL2, WL3, WL4]
STF 01:37:33.046 - |
STF 01:37:33.046 - Monitoring processes: WL1 WL2 WL3 WL4
STF 01:37:33.349 - **FAILED** Process WL4 ended with exit code (1) and not the expected exit code/s (0)
STF 01:37:33.350 - Monitoring Report Summary:
STF 01:37:33.350 -   o Process WL1 is still running
STF 01:37:33.350 -   o Process WL2 is still running
STF 01:37:33.350 -   o Process WL3 is still running
STF 01:37:33.350 -   o Process WL4 ended with exit code (1) and not the expected exit code/s (0)

68.WL4.stderr

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_1/63/ SharedClassesAPI_

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_1/63/system_test_output.tar.gz

STF 21:58:29.232 - +------ Step 69 - DefaultLocUtilitiesJVMTI: Monitor workload processes
STF 21:58:29.232 - | Wait for processes to meet expectations
STF 21:58:29.232 - |   Processes: [WL1, WL2, WL3, WL4]
STF 21:58:29.232 - |
STF 21:58:29.232 - Monitoring processes: WL1 WL2 WL3 WL4
STF 21:58:29.655 - **FAILED** Process WL2 ended with exit code (1) and not the expected exit code/s (0)
STF 21:58:29.656 - Monitoring Report Summary:
STF 21:58:29.656 -   o Process WL1 is still running
STF 21:58:29.656 -   o Process WL2 ended with exit code (1) and not the expected exit code/s (0)

66.WL2.stderr

JVMSHRC254E Error setting length of shared class cache file
JVMSHRC336E Port layer error code = -105
JVMSHRC337E Platform error message: (112) There is not enough space on the disk.
JVMSHRC231E Error setting shared class cache file length
JVMSHRC336E Port layer error code = -105
JVMSHRC337E Platform error message: (112) There is not enough space on the disk.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
pshipton commented 3 years ago

@AdamBrousseau disabled https://openj9-jenkins.osuosl.org/computer/win2012-x86-6/

AdamBrousseau commented 3 years ago

I cleared a few GB. These machines all seem to be sitting around 95%+ capacity on the C:\ drive. The Jenkins workspace is on F:\ but I assume the SCC is getting created in C:\tmp.

pshipton commented 3 years ago

Re-enabled the machine.

pshipton commented 2 years ago

Out of space problem on https://openj9-jenkins.osuosl.org/computer/win2012-x86-8 Disabled the machine until it can be fixed. @AdamBrousseau

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.system_x86-64_windows_Nightly_testList_1/193/ https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Release_testList_1/17/

JVMSHRC254E Error setting length of shared class cache file
JVMSHRC336E Port layer error code = -105
JVMSHRC337E Platform error message: (112) There is not enough space on the disk.
JVMSHRC231E Error setting shared class cache file length
JVMSHRC336E Port layer error code = -105
JVMSHRC337E Platform error message: (112) There is not enough space on the disk.
JVMSHRC840E Failed to start up the shared cache.
pshipton commented 2 years ago

win2012-x86-4 :

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/209 - SharedClassesAPI_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/209/system_test_output.tar.gz

66.WL2.stderr

JVMSHRC254E Error setting length of shared class cache file
JVMSHRC336E Port layer error code = -105
JVMSHRC337E Platform error message: (112) There is not enough space on the disk.
JVMSHRC231E Error setting shared class cache file length
JVMSHRC336E Port layer error code = -105
JVMSHRC337E Platform error message: (112) There is not enough space on the disk.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/155/ java/lang/StringBuffer/HugeCapacity.java JVMJ9VM015W Initialization error for library j9gc29(5): Failed to allocate default memory space. java/util/StringTokenizer/FormFeed.java Error. Cannot get VM for test: java.net.SocketTimeoutException: Accept timed out

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.system_x86-64_windows_Nightly_testList_1/1/ SharedClassesAPI_1

pshipton commented 2 years ago

Not sure why these problems occur periodically. Sanitize and cleanup seem to fix them, so the problems don't persist.

I managed to get win2012-x86-8 back online.

pshipton commented 2 years ago

Another issue on win2012-x86-8

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/211 SharedClassesAPI_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/211/system_test_output.tar.gz

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
AdamBrousseau commented 2 years ago

There was a 2.5G core in ~/ I cleaned up. Hopefully that helps. Page disk is full also. I will look into that.

pshipton commented 2 years ago

Another on win2012-x86-4

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/213/ SharedClassesAPI_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/213/system_test_output.tar.gz

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
AdamBrousseau commented 2 years ago

Set the page file to delete on reboot. Didn't seem to do anything though. https://www.techrepublic.com/article/how-to-delete-the-windows-10-paging-file-on-every-shut-down/

pshipton commented 2 years ago

Another on win2012-x86-4 Guessing it needs some manual cleanup.

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/214

pshipton commented 2 years ago

Deleted a 135MB core file from /home/jenkins present since Oct 7.

There is a 2GB pagefile.sys on the D: drive, and no room for anything bigger.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
C:/cygwin64     100G   98G  2.6G  98% /
D:              2.0G  2.0G   32M  99% /cygdrive/d
F:               45G  4.4G   40G  10% /cygdrive/f
pshipton commented 2 years ago

In /cygdrive/f/Users/jenkins there was a 1GB core file and other smaller diagnostic artifacts from Dec 15.

pshipton commented 2 years ago

Deleted jdk16 from bootjdks.

pshipton commented 2 years ago

Found and deleted an old obsolete 300MB shared cache from 2019 in /cygdrive/c/Documents\ and\ Settings/jenkins/Local\ Settings/Application\ Data/javasharedresources

pshipton commented 2 years ago

When I look at win2012-x86-8, it's got a 8GB drive for the pagefile, which is more reasonable. 2GB is too small, the pagefile drive on win2012-x86-4 needs to be fixed. I thought we fixed the pagefiles on all the machines some time ago.

D:              7.9G  7.9G   32M 100% /cygdrive/d
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_0/161/ - win2012-x86-6 SharedClassesAPI_0

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/222/ - win2012-x86-4 SharedClassesAPI_0

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_1/167/ - win2012-x86-4 https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_1/168/ - win2012-x86-4 SharedClassesAPI_1

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_1/170 - win2012-x86-4 SharedClassesAPI_1

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.system_x86-64_windows_Nightly_testList_1/227/ - win2012-x86-7 SharedClassesAPI_1

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Filesystem      Size  Used Avail Use% Mounted on
C:/cygwin64     100G   99G  2.0G  99% /
D:              7.9G  7.9G   32M 100% /cygdrive/d
F:               45G  1.9G   43G   5% /cygdrive/f

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/226 - win2012-x86-7 SharedClassesAPI_0

pshipton commented 2 years ago

Machine problems on win2012-x86-5. There were similar issues yesterday as well. This machine needs a reboot.

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.openjdk_x86-64_windows_Nightly/11/ java/lang/ProcessHandle/TreeTest.java.TreeTest java/lang/ProcessHandle/TreeTest.java.TreeTest java/lang/StringBuffer/HugeCapacity.java.HugeCapacity java/lang/StringBuffer/HugeCapacity.java.HugeCapacity java/util/Properties/StoreReproducibilityTest.java.StoreReproducibilityTest java/util/regex/PatternStreamTest.java.PatternStreamTest

pshipton commented 2 years ago

Maybe related? Test on win2012-x86-6 Terminating on signal SIGTERM(15)

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.system_x86-64_windows_Nightly_testList_1/228 SharedClasses.SCM23.MultiThread_1 -Xcompressedrefs -Xjit -Xgcpolicy:gencon

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.system_x86-64_windows_Nightly_testList_1/229/ - win2012-x86-7 SharedClassesAPI_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk8_j9_extended.system_x86-64_windows_Nightly_testList_1/229/system_test_output.tar.gz

67.WL3.stderr

JVMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_x86-64_windows_Nightly/229/ - win2012-x86-5 java/lang/ProcessBuilder/Basic.java

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_1/228 - win2012-x86-6 SharedClassesAPI_1

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_windows_Nightly_testList_0/173/ - win2012-x86-7 SharedClassesAPI_0

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/175/ - win2012-x86-2 java/lang/StringBuffer/HugeCapacity.java

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.openjdk_x86-64_windows_Release/3/ - win2012-x86-5 java/lang/ProcessBuilder/Basic.java#id0.Basic_id0 java/lang/ProcessHandle/TreeTest.java.TreeTest java/lang/ProcessHandle/TreeTest.java.TreeTest java/lang/StringBuffer/HugeCapacity.java.HugeCapacity java/lang/StringBuffer/HugeCapacity.java.HugeCapacity

pshipton commented 2 years ago

https://github.com/eclipse-openj9/openj9/issues/14674 - win2012-x86-2 https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.openjdk_x86-64_windows_Nightly/14/ java/lang/String/StringRepeat.java#id1

pshipton commented 2 years ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/229/ - win2012-x86-6 SharedClassesAPI_0

JasonFengJ9 commented 1 year ago

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_x86-64_windows_Nightly_testList_0/487/ (win2012-x86-6)

variation: Mode110
JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

SharedClassesAPI_0_FAILED

aqa-tests/TKG/output_16782664088475/SharedClassesAPI_0/20230308-035108-SharedClassesAPI/results/63.WL4.stderr

VMSHRC659E An error has occurred while opening shared memory
JVMSHRC662I Error recovery: destroyed semaphore set associated with shared class cache.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.