search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

-Xtrace & Method Enter/Exit Hooks Compiler Support for CRIU #17385

Open dsouzai opened 1 year ago

dsouzai commented 1 year ago

Background

There are two different types of method enter/exit tracing available:

  1. -Xtrace:print=mt,methods=...
  2. JVMTI method enter/exit hooks

JIT Implementation

If either of these types of tracing are possible, then the ILGenerator generates IL containing TR::MethodEnterHook and/or TR::MethodExitHook op codes. Later on in the compilation, TR_J9VMBase::lowerMethodHook is invoked on these op codes.

If -Xtrace is specified on a particular method, then the compiler generates a call to the appropriate helper [1]. Otherwise, if the JVM may have to report method enter/exit hooks to a JVMTI agent, then a runtime test is generated [2].

AOT Implementation

Like the JIT case, if either of these types of tracing are possible, then the ILGenerator generates IL containing TR::MethodEnterHook and/or TR::MethodExitHook op codes, on which TR_J9VMBase::lowerMethodHook is invoked. The difference though is that regardless of the type of tracing, the compiler generates a TR_MethodEnterExitGuard [3]. At AOT load time, if either type of tracing is possible, the guard is activated; otherwise, it is patched to be a NOP.

CRIU Support Proposal

At the moment, if tracing is specified post-restore, all code in the code cache is invalidated. The following is a proposal to minimize the performance impact of such an action.

1. Common Implementation

Because the AOT implementation is different, it complicates any solution - even if we decide to never generate any AOT code pre-checkpoint, the actual source code complexity is still going to be much higher. Therefore, the first step is to have a common implementation. This is relatively straightforward to do:

  1. Delete code that generates TR_MethodEnterExitGuard.
  2. Add a flag to every inlined method validation to indicate if it was specified in the -Xtrace filter.
  3. Add flags to the AOT Method Header to
    1. Indicate if the top most method (i.e. the method being compiled) was specified in the -Xtrace filter
    2. Indicate whether the method contains the runtime tests for the JVMTI callbacks.
  4. Add a relocation for runtime test for the JVMTI callbacks.

In both this implementation and the existing implementation, the only way AOT code could report to the VM on method enter/exit is if it knew it had to at compile time. The main difference is that in the existing implementation, if the code had the guard, it would be patched to not report to the VM whereas in the proposed implementation, it would continue to report to the VM (in the case of -Xtrace) or continue to have the runtime test (in the case of the JVMTI callback) thus increasing the path length. However, I don't the added complexity of having a completely different implementation for AOT is justified to prevent this extra path length considering that it will only happen if one generates AOT with the tracing and then removes the option in a subsequent run.

2. Additional Runtime Tests Pre-Checkpoint

There are a few different approaches to take regarding runtime tests:

a. Always generate the JVMTI runtime tests; invalidate only the methods that match the -Xtrace filter

Conceptually this is the simplest solution. However, the complexity comes entirely from inlined methods. Often times, inlined methods can have their guards removed because they've been merged with some previous guard. I looked at TR_BreakpointGuard, but those are only generated for direct calls; I guess the involuntary OSR mechanism handles everything else. If there's a good way to ensure we can invalidate any inlined method, then this approach should be considered, both because of the simplicity in both concept and the code generated.

b. Generate JVMTI runtime tests as well as -Xtrace filter tests

This would involve generating code to do what methodBeingTraced [4] does in addition to the the JVMTI runtime tests. The JVMTI runtime tests would only execute if -Xtrace is not enabled. However, the logic in methodBeingTraced is more involved than a simple test and so it might get quite expensive if it's done twice for every method. Specifically, methodBeingTraced:

  1. Tests if -Xtrace is enabled
  2. Fetches the fetchMethodExtendedFlagsPointer which involves probably multi-instruction arithmetic [5]
  3. Tests if this method matches the -Xtrace filter via J9_RAS_IS_METHOD_TRACED

c. Generate JVMTI runtime tests as well as only whether -Xtrace is enabled

This would only involve generating the first test (i.e. test if -Xtrace is enabled) along with the JVMTI runtime tests. If -Xtrace is enabled, then we jump to the VM; otherwise we perform the JVMTI runtime test. This approach would result in smaller code size, but if -Xtrace is enabled, we will end up calling in the VM for every method enter/exit, which is probably no better than the previous approach.

[1] https://github.com/eclipse-openj9/openj9/blob/45ed10ad1d3e751f1116ba846459d2adfd311c65/runtime/compiler/env/VMJ9.cpp#L2980 [2] https://github.com/eclipse-openj9/openj9/blob/45ed10ad1d3e751f1116ba846459d2adfd311c65/runtime/compiler/env/VMJ9.cpp#L3076 [3] https://github.com/eclipse-openj9/openj9/blob/45ed10ad1d3e751f1116ba846459d2adfd311c65/runtime/compiler/env/VMJ9.cpp#L3000 [4] https://github.com/eclipse-openj9/openj9/blob/45ed10ad1d3e751f1116ba846459d2adfd311c65/runtime/oti/VMHelpers.hpp#L1175-L1186 [5] https://github.com/eclipse-openj9/openj9/blob/45ed10ad1d3e751f1116ba846459d2adfd311c65/runtime/util/extendedmethodblockaccess.c#L28-L35

dsouzai commented 1 year ago

@vijaysun-omr what are your thoughts?

dsouzai commented 1 year ago

If there's a good way to ensure we can invalidate any inlined method, then this approach should be considered, both because of the simplicity in both concept and the code generated.

I was thinking about this, and it occurred to me that maybe we could leverage the TR_MethodEnterExitGuard. Currently it's used for the AOT implementation, but we could repurpose it to essentially disable inlined methods. We can guarantee it'll always get generated because of lowerMethodHook. I'll look into this and see how feasible it is.

vijaysun-omr commented 1 year ago

I think what you are planning next seems reasonable to me. Let us discuss other options only after you have evaluated the code complexity of using the TR_MethodEnterExitGuard as you are describing first. More importantly the performance of the runtime tests for JVMTI in all code compiled before checkpoint may also need to be evaluated but maybe the fact that this is a fairly small set of methods in the full scheme of things means that the steady state impact won't be substantial.