Quarkus 3.11: native executable has almost twice less throughput than JVM one

[nicolasduminil]

Describe the bug

Hello, I'm trying to compare the number of requests per second of a simple application running in JVM mode with the same one running as a Linux native executable.

Expected behavior

I'd expect that the number of requests per second of the application running in native executable mode be much higher than the one running in JVM mode.

Actual behavior

Surprising enough, the application running in JVM mode shows twice more requests per second.

How to Reproduce?

Reproducer: https://github.com/nicolasduminil/quarkus-simple

To reproduce:

Run in JVM mode:

$ git clone https://github.com/nicolasduminil/quarkus-simple
$ cd quarkus-siple
$ mvn -DskipTests package
$ ...
$ java -jar target/quarkus-app/quarkus-run.jar 

In another terminal run:

$ wrk -t10 -c50 -d40s http://localhost:8080/time
Running 40s test @ http://localhost:8080/time
  10 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.17ms    9.95ms 243.44ms   99.37%
    Req/Sec    12.42k     3.29k   45.56k    84.13%
  4920561 requests in 40.10s, 469.26MB read
Requests/sec: 122707.89
Transfer/sec:     11.70MB

Now, run in native mode:

$ mvn -DskipTests -Dnative -Dquarkus.native.container-build install
$ ...
$ target/quarkus-simple-1.0-SNAPSHOT-runner

Now, in another terminal run:

$ wrk -t10 -c50 -d40s http://localhost:8080/time
Running 40s test @ http://localhost:8080/time
  10 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.07ms    1.65ms  72.33ms   92.48%
    Req/Sec     6.80k   828.46    21.98k    75.39%
  2706499 requests in 40.10s, 258.11MB read
Requests/sec:  67495.08
Transfer/sec:      6.44MB

So, the number of requests per second is of 122707.09 in JVM mode and of 64975.08 in native mode.

Is that normal and, if yes, why ?

Output of uname -a or ver

Linux nicolas-XPS-15-9570 5.15.0-116-generic #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

java version "21.0.3" 2024-04-16 LTS Java(TM) SE Runtime Environment (build 21.0.3+7-LTS-152) Java HotSpot(TM) 64-Bit Server VM (build 21.0.3+7-LTS-152, mixed mode, sharing)

Mandrel or GraalVM version (if different from Java)

quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21

Quarkus version or git rev

3.11

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546) Maven home: /opt/apache-maven-3.9.5 Java version: 21.0.3, vendor: Oracle Corporation, runtime: /usr/lib/jvm/jdk-21-oracle-x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.15.0-116-generic", arch: "amd64", family: "unix"

Additional information

N/A

[quarkus-bot[bot]]

/cc @Karm (mandrel), @galderz (mandrel), @zakkak (mandrel,native-image)

[zakkak]

Hello @nicolasduminil

I'd expect that the number of requests per second of the application running in native executable mode be much higher than the one running in JVM mode.

Why do you expect that? Please have a look at this detailed answer on a similar issue https://github.com/oracle/graal/issues/979#issuecomment-480786612

To better understand what's happening specifically in your case you may have a look at https://github.com/quarkusio/quarkus/blob/main/TROUBLESHOOTING.md#troubleshooting-performance-issues

Hope that helps

[nicolasduminil]

@zakkak Many thanks for sending me links to how to profile the code but this wasn't at all my point. At least you're making clear that running native executable is much slower (twice) than running in JVM. It's not very intuitive that interpreting the code might be faster than executing native code, that's why I (and Im not the only one) was expecting a very different result. But it's good to know anyway. AFAIC, if the only benefit of native executable is the startup time and the footprint for the price of much less throughput, then not for me.

[zakkak]

It really depends on the application and your needs (higher throughput, higher throughput per watt, faster start up, less energy consumption, etc.). If your benchmarks indicate that native executables are not covering your needs, or that the trade offs are not worth it, then you are probably right. Happy that I was able to help.

[galderz]

... It's not very intuitive that interpreting the code might be faster than executing native code...

Maybe you need to brush up a bit on HotSpot, in particular the JIT ;)