Karm
commented
10 months ago Thank you so much @roberttoyonaga :partying_face: :fireworks:, This is an awesome addition. I will read the source, try it, and share feedback. Right off the bat, let me respond to some remarks in your PR text:
sudo. This is not a problem on my local machine...
Sudo is not a problem for CI either. E.g. podman tests also require sudo so as to be able to attach debuggers to container on RHEL 8 etc. Some care needs to be taken about putting the system in its original state though as not all test systems are throwaway VMs. We execute also on long running baremetal systems. When we know about it now, we can easily make sure it's O.K.
Client server sharing resources But this problem will need to be addressed if we want to be confident we have reliable results that can be published.
Mandrel-integration-tests is by no means a vehicle for reliable, meaningful performance testing as its inherent design flaw, if you will, is that it disturbs the tested system by all JVM+Maven runtime shenanigans. We cannot use it to confidently publish results about JFR monitored apps throughput in absolute numbers for a given system. We can and we will use it heavily to compare Mandrel versions/Quarkus versions and to guard against regressions between versions though.
In the future I might spend some more time tuning this test using different events so that it can be used with 22.3 and 21.2.
23.0 support is splendid. 22.3 would be nice to have. 21.x is superfluous as we are retiring that version support from the TS soon #178 .
roberttoyonaga
Summary
This PR attempts to add a JFR performance test that checks RSS upon start up, image size, start up time, mean response time, and latency percentiles (50, 90, 99). The test uses Hyperfoil to benchmark response times and a slightly altered Quarkus getting-started quickstart that has been rigged and tuned to produce JFR events.
I've been running it with
mvn clean verify -Ptestsuite -DexcludeTags=all -DincludeTags=jfr-perfDetails
Test runs
There should be 4 test runs in total: No JFR + "normal" case benchmark, Yes JFR + "normal" case benchmark, No JFR +"worst" case benchmark, Yes JFR+"worst" case benchmark. The goal of the test is not to obtain results to analyze absolutely, but instead for comparison purposes. The report.md and measurments.csv files should also contain "diff" info for convenience.
Hyperfoil
Hyperfoil is downloaded from quay and run in "standalone" mode. The test communicates with the Hyperfoil controller via its REST API. Two Hyperfoil benchmarks have been added in this PR, a "normal" case and "worst" case. The "worst" case benchmark is meant to emphasize the performance impact of JFR infrastructure, and it emits an unrealistically large number of events. The "normal" case benchmark is meant to be more realistic and is what will be used to generate the metrics that are compared to thresholds defined in threshold.properties. Please see the test's javadoc for more info.
Quarkus app
The app has two main endpoints. "hello/work/{text}" is what the "worst" case benchmark hits, and "hello/regular/{text}" is what the normal case benchmark hits. This requires the test to only build the app twice (with and without JFR included in the image).
Worth mentioning
Cache and turbo boost
The test clears the cache before measuring startup time. This is averaged over several trials. Turbo boost is disabled before running the Hyperfoil benchmark and enabled again afterwards. Both of these operations require executing a command with
sudo. This is not a problem on my local machine, because the test pauses midway and prompts me to enter my passcode. However, I'm not sure if this will be an issue if this test suite is run automatically as part of CI. If this is a problem, I'll instead add some "warm up" runs to populate the cache before observing. I'm not sure how else to deal with turbo boost though.Client server sharing resources
The hyperfoil benchmark part of the test uses a client and server. Currently, they run on the same machine with out any isolation of resources. This maybe won't be an issue if this perf test is going to be used as a "sanity check" to give us a rough idea of how JFR is performing. But this problem will need to be addressed if we want to be confident we have reliable results that can be published.
Mandrel version
Currently the test is only set up to run with 23.0. This is because that's the first version
jdk.ThreadParkis supported. This event is useful to test the JFR infrastructure because we can manually invoke an event emission by callingLockSupport.parkNanos(). A similar thing can be achieved withjdk.ThreadSleepand callingThread.sleep(), orjdk.JavaMonitorWaitandwait(). However, even when the specified sleep/wait times are set to 1ns, the non-JFR overhead of calling these methods is actually relatively large. By contrast,LockSupport.parkNanos()has very low non-JFR overhead, which allows us to spend a higher percentage of time exercising the JFR code.In the future I might spend some more time tuning this test using different events so that it can be used with 22.3 and 21.2.
Profiling with Perf
Here I basically just want to show that the Quarkus app is in fact spending most of its time where we'd expect it to (dealing with the thread park events).
With JFR: Built with
mvn package -Pnative -Dquarkus.version=3.1.2.Final -Dquarkus.native.debug.enabled -Dquarkus.native.monitoring=jfr -Dquarkus.native.additional-build-args=-H:+SignalHandlerBasedExecutionSampler,-H:-DeleteLocalSymbolsWithout JFR Built with
mvn package -Pnative -Dquarkus.version=3.1.2.Final -Dquarkus.native.debug.enabled -Dquarkus.native.additional-build-args=-H:-DeleteLocalSymbols