Reduce GC Unloading Times During Heavily DLT Optimized workloads

[AlenBadel]

Background

J9Method_HT This is a part of the DLT Optimization. The Dynamic Loop Transformation attempts to identify a selection of candidate methods to be optimized. Keeping in mind the structure of DLTTracking[1]

DLTRecord Similarly, to the J9Method_HT this table holds entries with references to J9Methods that have been DLT Compiled.

Class Unloading

Currently during class unloading, the GC generates a list of dying class-loaders and then calls MM_ClassLoaderManager::cleanUpClassLoadersStart. Here, all classes that belong to that class-loader are then marked as dying classes[2]. VM Hooks are then issued to allow the JIT[3], and VM[4] to execute required unloading work before the memory is finally freed.

Before we dive further into the JIT specifics, we need to address the special case of unload an anonymous class. Dying anon classes are not unloaded with their respective class-loader. All other classes are always unloaded with their class-loader. [5]

The JIT, in turn subscribes to TRIGGER_J9HOOK_VM_CLASS_LOADER_UNLOAD, and TRIGGER_J9HOOK_VM_ANON_CLASSES_UNLOAD VM hooks. During these hooks, the J9Method_HT, and the DLTRecord Table are purged of any entry which holds a J9Method reference that belongs to a dying class[6].

As we can see in [3], that a TRIGGER_J9HOOK_VM_CLASS_LOADER_UNLOAD hook is issued on each dying class-loader, and for each unloaded class-loader we need to iterate the whole table to remove all entries that belong to dying classes. Since each dying class-loader can be identified from it's flags via classLoader->gcFlags multiple iterations are not necessary. Hence the JIT can remove all entries that contain method references which belong to dying classes in one iteration to decrease GC unloading times.

Implementation:

• JIT is to subscribe to TRIGGER_J9HOOK_VM_CLASS_LOADERS_UNLOAD[4] which is only called once by the GC. This hook is subscribed inside HookedByTheJIT. A new routine is created to just clean up the DLTRecord, and J9Method_HT tables.
• onClassUnloading[7] will need to be modified to not accept any arguments, since we can just check for classLoader->gcFlags |= J9_GC_CLASS_LOADER_DEAD;(Dead Class-loaders) or clazz->classDepthAndFlags |= J9AccClassDying;(Anon Classes)
• Finally, the clean-up code that is invoked during TRIGGER_J9HOOK_VM_CLASS_LOADER_UNLOAD in [6] can be removed. Invoking the clean-up routines within TRIGGER_J9HOOK_VM_ANON_CLASSES_UNLOAD will not be removed because again Anon classes are not unloaded with their class loaders.

[1] https://github.com/eclipse/openj9/blob/27e93e6cae493f2a3945528330adf5faacec3e4b/runtime/compiler/control/CompilationRuntime.hpp#L316-L329 [2] https://github.com/eclipse/openj9/blob/11f2b723867955926f2f10d545c90d708a304c63/runtime/gc_base/ClassLoaderManager.cpp#L330-L341 [3] https://github.com/eclipse/openj9/blob/11f2b723867955926f2f10d545c90d708a304c63/runtime/gc_base/ClassLoaderManager.cpp#L586-L587 https://github.com/eclipse/openj9/blob/c11ff942dc4ca0b600f2860ddc70548e7640aa7b/runtime/vm/jvmfree.c#L361 https://github.com/eclipse/openj9/blob/11f2b723867955926f2f10d545c90d708a304c63/runtime/gc_base/ClassLoaderManager.cpp#L343-L353 [4] https://github.com/eclipse/openj9/blob/11f2b723867955926f2f10d545c90d708a304c63/runtime/gc_base/ClassLoaderManager.cpp#L355-L359 [5] https://github.com/eclipse/openj9/blob/11f2b723867955926f2f10d545c90d708a304c63/runtime/gc_base/ClassLoaderManager.cpp#L315-L325 [6] https://github.com/eclipse/openj9/blob/02172866b19c2130f65dead89ee979f71ea01cad/runtime/compiler/control/HookedByTheJit.cpp#L2655-L2659 https://github.com/eclipse/openj9/blob/02172866b19c2130f65dead89ee979f71ea01cad/runtime/compiler/control/HookedByTheJit.cpp#L2823-L2827 [7] https://github.com/eclipse/openj9/blob/58e954e558f264b90b9ec1dc08a2c158abf5eb6e/runtime/compiler/control/CompilationThread.cpp#L12558

[AlenBadel]

I've created a draft of this enhancement. See https://github.com/AlenBadel/openj9/commit/10ce02db8bff98dbd73ab5a926ab1355ae128edb

Will be continuing to testing.

[AlenBadel]

Closing since the PR was merged.

@DanHeidinga Feel free to re-open this if you would like to see an experimental proof of any speed-up.

[DanHeidinga]

Feel free to re-open this if you would like to see an experimental proof of any speed-up.

It would be good to quantify the speed-up as class unloading pauses are one of the issues that periodically show up as service issues.

@dmitripivkine are there any existing tests that stress class unloading that Alen can use to benchmark this change?

[AlenBadel]

Getting class unloading pause times is quite simple. We have a test that heavily causes class unloading. https://github.com/eclipse/openj9/blob/266e5b614ee2dd7110db81c1d2ea310ef5b95399/test/functional/cmdLineTests/gcRegressionTests/src/com/ibm/tests/garbagecollector/TestClassLoaderMain.java

The only issue is the commit addresses speed-up during heavily DLT compiled workloads, something I don't think we have. We can certainly try creating one.

[dmitripivkine]

Feel free to re-open this if you would like to see an experimental proof of any speed-up.

It would be good to quantify the speed-up as class unloading pauses are one of the issues that periodically show up as service issues.

@dmitripivkine are there any existing tests that stress class unloading that Alen can use to benchmark this change?

The test we have (mentioned above) does stressed class unloading but it never helps with performance investigation/verification for issue like this. I believe it is because JIT tables have small size and it is hard to measure walk time difference.

[AlenBadel]

@andrewcraik Would you have an idea of how to increase the frequency of DLT Compilations?

[andrewcraik]

I don't think that it is easy to do because the loop needs to be long running to start with - @mprivu might have some ideas, but it will be hard to get loads of methods DLT'd