Ability to provide test image in addition to JDK/JRE packages

[sophia-guo]

Native tests need executables or shared libraries. To enable those tests test image need to be built and accessible.

Comments from @pshipton "The executables are already created as part of the OpenJ9 build, but the OpenJ9.gmk [1] doesn't copy them into the sdk or jre images (which is correct). They should be copied into the test image. There will be changes required to keep the test image around so it can be used for testing. Currently we only keep the sdk image.

To extend on this, the test libraries should also be extracted from the vm directory and put into the test image. This will require fixes to the tests so they can use the shared libraries found in the test image instead of getting them "for free" as part of the VM."

[llxia]

Note: #2761 addressed the issue for having separate test image, but we are still lacking some native test exe: such as algotest, jsigjnitest, etc. https://github.com/ibmruntimes/openj9-openjdk-jdk11/blob/98f20388ae61f5df0da85e3d360d14ab29c3279a/closed/OpenJ9.gmk#L242

[sophia-guo]

I believe current separate test image collect all test related shared libraries. Could we do same for native tests related executables, which are already created as part of the Openj9 build. Could just copy them to test images too? So we can re-enable a bunch of native tests? @pshipton @keithc-ca

My lists of executables are: algotest cfdump ctest pltest shrtest gc_rwlocktest glaunch dyntest vmtest vmLifecyleTests thrstatetest testjep178_static testjep178_dynamic jsigjnitest invtest

Which I could see built under vm folder.

[keithc-ca]

That seems reasonable; I'll put some PRs together.

[keithc-ca]

The PRs also include bcvunit in the list of test executables.

[pshipton]

@sophia-guo I've merged the changes for Java 11/12. I've also asked Keith to backport this to Java 8 so the executables (and test libraries) will be available there as well.

[sophia-guo]

I tried some native test. And most of them got the message: error while loading shared libraries: libj9prt29.so/libj9thr29.so/libjsig.so: cannot open shared object file: No such file or directory with export LD_LIBRARY_PATH=""/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../../native-test-libs/:/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../lib/compressedrefs:/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../lib/j9vm"

Checked libj9prt29.so/libj9thr29.so/libjsig.so are under /home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../lib/compressedrefs @keithc-ca any suggestion? Thanks.

13:28:18 Running test algotest2_0 ...
13:28:18 ===============================================
13:28:18 algotest2_0 Start Time: Tue Sep 25 17:28:15 2018 Epoch Time (ms): 1537896495959
13:28:18 test with NoOptions
13:28:18 { export LD_LIBRARY_PATH=""/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../../native-test-libs/:/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../lib/compressedrefs:/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../lib/j9vm":"; \
13:28:18 '/home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../../native-test-libs//algotest' -avltest:/home/j9build/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/functional/NativeTest/../algotest/avltest.lst -Xcheck:memory:ignoreUnfreedCallsite=zip/:dbgwrapper:unknown:wrapper; \
13:28:18 if [ $? -eq 0 ] ; then echo ""; echo "algotest2_0""_PASSED"; echo ""; else echo ""; echo "algotest2_0""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/home/j9build/workspace/Grinder_Sandbox/openjdk-tests/TestConfig/scripts/testKitGen/../../../TestConfig/test_output_15378964951391/TestTargetResult";
13:28:18 /home/j9build/workspace/Grinder_Sandbox/openjdkbinary/j2sdk-image/bin/../../native-test-libs//algotest: error while loading shared libraries: libj9prt29.so: cannot open shared object file: No such file or directory
13:28:18 
13:28:18 algotest2_0_FAILED

[keithc-ca]

There's some cleanup with quotes that might help: Notice the double "" at the beginning of LD_LIBRARY_PATH and the ":" at the end, but I'm not convinced that causes your problems.

Assuming everything is where you expect it to be, it should work as it does for me:

$ pwd
/home/keithc/space/jdk11/build/linux-s390x-normal-server-release/images/test/openj9
$ LD_LIBRARY_PATH=../../jdk/lib/compressedrefs ldd algotest
    libj9prt29.so => ../../jdk/lib/compressedrefs/libj9prt29.so (0x000003ff9dc80000)
    libj9thr29.so => ../../jdk/lib/compressedrefs/libj9thr29.so (0x000003ff9dc00000)
    libm.so.6 => /lib/s390x-linux-gnu/libm.so.6 (0x000003ff9da80000)
    libpthread.so.0 => /lib/s390x-linux-gnu/libpthread.so.0 (0x000003ff9da00000)
    libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ff9d800000)
    libdl.so.2 => /lib/s390x-linux-gnu/libdl.so.2 (0x000003ff9d780000)
    libomrsig.so => /home/keithc/space/jdk11/build/linux-s390x-normal-server-release/images/test/openj9/../../jdk/lib/compressedrefs/libomrsig.so (0x000003ff9d700000)
    librt.so.1 => /lib/s390x-linux-gnu/librt.so.1 (0x000003ff9d680000)
    /lib/ld64.so.1 (0x000003ff9dd00000)

[sophia-guo]

Thanks @keithc-ca , looks like I've got a wrong jdk. Will double check it.

[sophia-guo]

@keithc-ca for windows 11 libs like j9thr29.dll are under the folder bin/compressedrefs instead of lib/compressedrefs. Is that expected? Thanks.

[keithc-ca]

Yes, on Windows, shared libraries are within the bin directory while on other platforms they're located within the lib directory. Windows has always been special that way.

[sophia-guo]

Tests have been enabled for 8 and 11. Close it.