Benchmark 9/19

[akarnokd]

See the most recent comparison in #7.

Comparative benchmark

• RxJava 1.2 (9/17)
• RxJava 2.0.0-DP0-SNAPSHOT (9/15)
• Reactor Core 3.0.3-BUILD.SNAPSHOT (9/19)

Environment:

• i7 4790 stock settings
• Windows 7 x64
• Java 8u102
• JMH 1.13
• -XX:MaxInlineLevel=20

Green is good, red is not so good.

• JMH-Compare-GUI workspace file with the data: https://gist.github.com/akarnokd/6550d42f4a4fb7bbead0ea3afdbbf42e

  Subscribe to a simple source

Just comparing 1.x and 2.x

Remarks:

• Subscribing to any type in 2.x has lower overhead than 1.x.

  Streaming multiple values (synchronous)

Remarks:

• There seems to be some extra overhead in the backpressured concatMapXRange (both 2.x and Reactor).
• Also flatMapJust has extra overhead for all 2.x types (could be unavoidable atomics).
• flatMapXRange 1M (which is flatMapJust 1M essentially) suffers the same overhead.
• 2.x Observable.flatten is not optimized yet and delegates to plain flatMap
• Otherwise, nice improvements all over the board.

  Asynchronous and pipelined streaming

Remarks:

• There seems to be some extra overhead or missing optimization from Flowable async 1.
• 2.x Observable lack of optimization (1-case) and likely buffer-bloat in longer streams.
• Generally not bad, but such benchmarks can be hectic (on Windows especially).

  Memory usage

Remarks:

• Generally half the memory usage.
• empty() seems to be occupying too much memory as if there 1M unique empty() instance.
• Behavior and Unicast seem to occupy double the memory for some reason.

  Subject/Processor synchronous streaming

Remarks:

• Async, Publish and small Replay is definitely faster in 2.x
• Larger Replay, any Unicast and any Behavior are half-to-quarter slower for some unknown reason.

  Blocking for an item (or completion)

Remarks:

• All seem to better at blockingly retrieve one element, except 2.x Observable for some reason.
• Reactor is optimized on blocking empty() retrieval

  Reactor 3 vs RxJava 2

• They have the same underlying architecture style.
• Operator internals are practically the same for most Flowable/Flux types.
• Reactor delegates some of its Mono operators to Flux operator internals.
• Reactor has more operators fusion-enabled, RxJava has most important operators fusion-enabled, its Observable has not been extensively optimized though.
• Reactor uses queue-suppliers, RxJava creates specific queues in operators.
• Reactor uses atomic field updaters, RxJava extends AtomicXXX classes but otherwise uses AtomicXXX instance fields.
• Reactor has only an onAssembly hook, RxJava has a subscribe-time hook and indirection on top of onAssembly hook.

  Subscribing

Remarks:

• Reactor uses a weaker form of backpressure in just and thus slightly better.

  Streaming synchronously

Remarks:

• Most operators are within the +/- 5% error margin.
• Reactor's flatMap seems to be slightly faster (5-15%)
• Reactor's flatten seems to be reasonably faster (1.5x to 2x).
• Reactor's iterable seems to be slightly slower (5-15%)

  Async streaming

Remarks:

• The async benchmark is a bit hectic, inconclusive which is better
• Reactor likes longer pipelined streams but dislikes shorter pipelined streams.

  Memory use

Remarks:

• Inconclusive, Reactor has more overhead in places and less in others.
• Reactor's processors are more compacted but leak API.
• Reactor has some drastic Replay overhead.

  Processor overhead

Remarks:

• Reactor is faster in general but slightly slower in large Replay and Unicast

  Blocking

Remarks:

• Reactor is optimized for blocking on an empty()

Remarks:

• Blocking for the first element is the same.

Remarks:

• Blocking for the last element is slower in Reactor.

  Conclusion

• RxJava 2.x is generally better in terms of performance and memory utilization than 1.x; most lower performing components can be optimized further.
• Reactor is generally 10-50% better in a Java 8 environment due to more inlined nature and even fewer allocations than RxJava 2.

[smaldini]

There's room for improvement in reactor I think given some latest design improvements but this is a massively complete report, congrats @akarnokd !

[daschl]

great info, that makes me even more wanting to move the couchbase SDK over to 2.x at some point, but still not sure how to figure out a smooth migration path :)

[ondreju93]

Hi How to understand the first table (Subscribe to a simple source) ? What are these compared values for RxJava 1.x and RxJava 2.x ? Are they time values? What is the unit? In the latter tables i can see information about count of operations. What is the count of operations in the first table?

[akarnokd]

@ondreju93

How to understand the first table (Subscribe to a simple source) ?

These are operations per second, larger is better tables. There is a baseline value chosen in each table and the other columns are measured relative to it. Green background indicate improvements, red mean worse performance.

What are these compared values for RxJava 1.x and RxJava 2.x ?

In the first table the, it shows how many times various, simple reactive types can be subscribed to in a second.

What is the count of operations in the first table?

There isn't one. They are simple sources with 0 or 1 element, depending on the type. Later tables feature longer sequences as those show how the infrastructure overhead is amortized over many items.

[ondreju93]

@akarnokd Thanks for the explanations Now everything is clear for me! Congrats for creating such a great benchmarks!