SC_Softmx_Increase_0 failures

[pshipton]

example failure: https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/17 for all failures see https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-All/620/

SC_Softmx_Increase_0 is failing on what appears to be all platforms and java versions after https://github.com/eclipse/openj9/pull/6536 is merged. This PR changes the format of the shared cache and consumes more space. I expect the test needs to be re-tuned.

STF 02:37:13.843 - +------ Step 6 - Print Shared Classes Cache Stats
STF 02:37:13.843 - | Print shared classes cache stats
STF 02:37:13.843 - |   Options:          -Xshareclasses:name=SCC_SysTest_Workload_Cache,cacheDir=/home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdk-tests/TestConfig/test_output_156654032010/SC_Softmx_Increase_0/20190823-023607-SharedClassesWorkloadTest_Softmx_Increase/results/cache${cacheOperation}
STF 02:37:13.843 - |   CacheName:        SCC_SysTest_Workload_Cache
STF 02:37:13.843 - |   CacheDir:         /home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdk-tests/TestConfig/test_output_156654032010/SC_Softmx_Increase_0/20190823-023607-SharedClassesWorkloadTest_Softmx_Increase/results/cache
STF 02:37:13.843 - |   CacheOperation:   ,printStats
STF 02:37:13.843 - |   ExpectedMessages: [Cache is 100% soft full]
STF 02:37:13.843 - |
STF 02:37:13.844 - Running command: /home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdkbinary/j2sdk-image/bin/java -Xshareclasses:name=SCC_SysTest_Workload_Cache,cacheDir=/home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdk-tests/TestConfig/test_output_156654032010/SC_Softmx_Increase_0/20190823-023607-SharedClassesWorkloadTest_Softmx_Increase/results/cache,printStats -Xcompressedrefs
STF 02:37:13.844 - Redirecting stderr to /home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdk-tests/TestConfig/test_output_156654032010/SC_Softmx_Increase_0/20190823-023607-SharedClassesWorkloadTest_Softmx_Increase/results/6.SCC.stderr
STF 02:37:13.844 - Redirecting stdout to /home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdk-tests/TestConfig/test_output_156654032010/SC_Softmx_Increase_0/20190823-023607-SharedClassesWorkloadTest_Softmx_Increase/results/6.SCC.stdout
STF 02:37:13.846 - Monitoring processes: SCC
SCC stderr 
SCC stderr Current statistics for cache "SCC_SysTest_Workload_Cache": 
SCC stderr 
SCC stderr Cache created with:
SCC stderr  -Xnolinenumbers                      = false
SCC stderr  BCI Enabled                          = true
SCC stderr  Restrict Classpaths                  = false
SCC stderr  Feature                              = cr
SCC stderr 
SCC stderr Cache contains only classes with line numbers
SCC stderr 
SCC stderr base address                         = 0x00007F4D7EC23000
SCC stderr end address                          = 0x00007F4D80000000
SCC stderr allocation pointer                   = 0x00007F4D7ECCEEC8
SCC stderr 
SCC stderr cache size                           = 20970912
SCC stderr softmx bytes                         = 1048576
SCC stderr free bytes                           = 1100
SCC stderr ROMClass bytes                       = 704200
SCC stderr AOT bytes                            = 40
SCC stderr Reserved space for AOT bytes         = -1
SCC stderr Maximum space for AOT bytes          = -1
SCC stderr JIT data bytes                       = 0
SCC stderr Reserved space for JIT data bytes    = -1
SCC stderr Maximum space for JIT data bytes     = -1
SCC stderr Zip cache bytes                      = 0
SCC stderr Startup hint bytes                   = 0
SCC stderr Data bytes                           = 142752
SCC stderr Metadata bytes                       = 7832
SCC stderr Metadata % used                      = 0%
SCC stderr Class debug area size                = 1662976
SCC stderr Class debug area used bytes          = 192044
SCC stderr Class debug area % used              = 11%
SCC stderr 
SCC stderr # ROMClasses                         = 168
SCC stderr # AOT Methods                        = 0
SCC stderr # Classpaths                         = 1
SCC stderr # URLs                               = 0
SCC stderr # Tokens                             = 0
SCC stderr # Zip caches                         = 0
SCC stderr # Startup hints                      = 0
SCC stderr # Stale classes                      = 0
SCC stderr % Stale classes                      = 0%
SCC stderr 
SCC stderr Cache is 99% soft full
SCC stderr 
SCC stderr Cache is accessible to current user = true
STF 02:37:14.053 - Monitoring Report Summary:
STF 02:37:14.054 -   o Process SCC ended with the expected exit code (1)
**FAILED** at step 6 (Print Shared Classes Cache Stats). Expected return value=0 Actual=0 at /home/jenkins/workspace/Test_openjdk13_j9_extended.system_x86-64_linux_Nightly/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_156654032010/SC_Softmx_Increase_0/20190823-023607-SharedClassesWorkloadTest_Softmx_Increase/execute.pl line 236.
STF 02:37:14.326 - **FAILED** execute script failed. Expected return value=0 Actual=1

[pshipton]

@Mesbah-Alam

[pshipton]

@Mesbah-Alam any outlook for getting this fixed? It's causing all the extended system testing to fail across all platforms.

[Mesbah-Alam]

I am looking at the test right now.

[Mesbah-Alam]

This test has 4 steps:

 *  ~~~~~~~~~~~~~~~~~~~~~~
 *  1) Start Jvm1 with options: "-XX:SharedCacheHardLimit=20m, -Xscmxsoftmx=1m". It should 
 *     create the shared classes cache and run a mini-mix load test and fill up the shared 
 *     classes cache.
 *  2) Once Jvm1 finishes its work, check cache stats to ensure cache is soft full. Fail if it isn't.
 *  3) Start Jvm2 that does -Xshareclasses:adjustsoftmx=2m. Check cache stats to ensure 
 *     cache's softmx limit has been increased to 2m.
 *  4) Start Jvm3 that runs the second mini-mix load. Make sure code is being written to 
 *     the increased space of the shared classes cache.

The test fails in step 2; i.e. it fills up to 99% when it's expected to fill up 100%. We need to retune the test.