Performance decrease w/Quarkus 1.5.0, Mongo Extension, Mutiny

[csotiriou]

It seems that there is a considerable throughput performance hit (approx 40%) when upgrading one microservice from an older stack using Quarkus to a newer one.

Older stack:

• Quarkus 1.2.1
• JDK 8 (running in OpenJ9)
• MongoDB extension (with Completion Stages)
• SOAP Calls (but reproducible with HTTP REST, without SOAP)

Newer stack:

• Quarkus 1.5.0
• JDK 11 (running in OpenJ9)
• MongoDB extension (with Mutiny)
• SOAP Calls (but reproducible with HTTP REST, without SOAP)

The microservice uses MongoDB for caching. Steps:

• Querying the DB for a key
  • If the key is found, return the value of this key from the MongoDB.
• If the key is not found, make an HTTP request
• Save the result to MongoDB. The result is saved as a model, using a Jackson codec.
• Return the result to the caller.

Our older stack was using RxJava, and was transforming the CompletionStages to Singles. The new stack uses Mutiny, and also exposes Mutiny Interfaces. In both cases, we use RestEasy interfaces, we are not using Reactive Routes.

On a performance load test lasting for approximately 30 minutes each, I have seen the following differences in performance:

The old stack's CPU load is something like the following

The new stack's behaviour is totally different, however.

It seems observed that the CPU consumption went up by 40%, thus losing throughput in the process. Both tests were conducted on a microservice using the same amount of RAM and CPU limits.

Tests already done to debug this:

• Switching to OpenJDK11 image. Tested with both the default garbage collector and the old one (-XX:UseParallelGC). The results were very similar (decreased performance).
• Switching the old implementation using Quarkus 1.2.1 to Java 11 (both OpenJ9). The performance was great (same as the one we had with the old implementation using Java 8)

[quarkusbot]

/cc @loicmathieu

[csotiriou]

Hello,

Just to add to this issue, I switched to the synchronous client in MongoDB, and I still faced problems. I decided to make another test:

• Use the synchronous MongoDB driver, but return it using Uni.createFrom().item(response) (just to expose Uni interfaces, for testing)
• Use the synchronous MongoDB driver again, but encapsulate it in RxJava's Single, to simulate as much of the codebase as the one we had before.

I was surprised to see that using RxJava, without any other change in the code, my performance returned back to normal.

I am led to believe that perhaps Mutiny itself (regardless of how it's used inside the Mongo extension) is the real issue here.

[gsmet]

There is an ongoing discussion with @johnaohara on Zulip. Hopefully, this will provide useful feedback.

[cescoffier]

Do you have context propagation enabled?

[csotiriou]

Yes, we use the smallrye context propagation dependency.

[cescoffier]

Thanks for the very quick response. @johnaohara, that can explain the overhead due to the numbers of interception.

[cescoffier]

(Even on a simple: Uni.createFrom(item) you have 2 interceptions.)

[csotiriou]

That would explain why even after ditching the asynchonous mongo driver still creates problems when even synchronous results are mixed with plain Uni.createFrom().

I would like to clarify that I always had context propagation enabled even with the old implementation that used RxJava (no performance issues there).

[cescoffier]

yes, but mutiny enables it automatically, RX does not (it requires an additional dependency).

[csotiriou]

I understand. Does this mean that if I was including the context propagation extra dependency for RXJava I would see the same performance hit as with mutiny?

Or this is something that can be improved on the side of mutiny/context propagation library ?

[cescoffier]

In the mutiny side, definitely - especially for Uni. It’s actually planned. It would be great to have a way to reproduce your case, so I can test the improvements.

[geoand]

Did we come to a conclusion on this one? Has it since been remedied?

[cescoffier]

We have done some improvements in Mutiny. The context-propagation issue is still valid I guess.

[geoand]

@csotiriou would it be possible to try Quarkus 1.12 and / or the Quarkus Snapshots to see if the issue still persists?

[csotiriou]

@geoand, it seems that the issue is fixed, I have experienced no more crashes and strange behaviour.

[geoand]

That's awesome, thanks!