Measure performance of SmartStream consumer for various scenarios

[nicholastmosher]

We want to investigate long pauses during certain SmartStream consumer scenarios. Specifically the following:

Scenario/Variables:

• Which SmartStream to use (constant)
• Cluster used: Local cluster (release mode)
• Client used
  • [x] CLI (release mode)
  • [x] fluvio-client-wasm (release mode)
  • [x] fluvio-client-wasm (debug mode)

[nicholastmosher]

Hey team! Please add your planning poker estimate with ZenHub @morenol @nacardin @sehz @simlay @tjtelan

[nicholastmosher]

Testing using the following (included in zip):

• SmartStream: fluvio_wasm_filter.wasm
• Contents: 100,000 random words from /usr/share/dict/words, approx 1Mb
• On git hash: 87a756b37fc49ad8261e3c11f86c8d5a59dac401

https://app.zenhub.com/files/205473061/cf7be662-5362-497f-b4a3-518f24ec2911/download

[nicholastmosher]

Scenario: Local Cluster, CLI (release mode)

Commands used:

To start the local cluster:

cargo build --bin fluvio --bin fluvio-run --release
target/release/fluvio cluster start --local

To produce words to topic:

target/release/fluvio produce consumer-test -f consumer-test/words.txt

To consume words with SmartStream, I made the following script, test-script.sh:

#!/usr/bin/env bash

head -n 61476 <(target/release/fluvio consume consumer-test -B -d --filter=consumer-test/fluvio_wasm_filter.wasm)

The reason I used head here is beccause there is a bug where -d does not work with SmartStream filters (#1504). To get around that I basically use head to read a fixed number of lines, the number of records I know come out of this stream when filtered, which I discovered through, uh, practical application of binary search.

For measurements, I ran hyperfine ./test-script.sh which gave the following output:

$ hyperfine ./test-script.sh
Benchmark #1: ./test-script.sh
  Time (mean ± σ):     379.2 ms ±  11.4 ms    [User: 32.1 ms, System: 58.5 ms]
  Range (min … max):   364.5 ms … 399.8 ms    10 runs

[nicholastmosher]

Scenario: Local Cluster, fluvio-client-wasm (release mode)

Set up cluster:

# In fluvio
$ cargo build --release --bin fluvio --bin fluvio-run
$ target/release/fluvio cluster start --local

Set up websocket proxy

# In fluvio-client-wasm
$ RUST_LOG=debug cargo run --manifest-path ./fluvio-websocket-proxy/Cargo.toml --target $(rustup show | grep 'Default host' | sed 's/Default host: //g')

Set up topic data:

$ fluvio topic create consumer-test
$ fluvio produce consumer-test -f consumer-test/words.txt // From test zip

Run wasm test:

$ wasm-pack test --firefox -- --release

Open firefox and in the console, note the measured test run time

Measurements:

• 4445ms
• 4322ms
• 4332ms
• 4475ms
• 4394ms
• 4344ms
• 4358ms
• 4337ms
• 4398ms
• 4336ms

Stats (in ms):

• min: 4322
• max: 4475
• mean: 4374.099999999999
• stddev: 49.65370076842213

[nicholastmosher]

From these results, I'm seeing that fluvio-client-wasm is certainly slower than the CLI, but not to the extent to cause the problems we were seeing before. I'm re-running the fluvio-client-wasm test in debug mode to see if it exhibits time more similar to the problems we have seen

[nicholastmosher]

Scenario: Same as above, but fluvio-client-wasm is in debug mode:

Measurements:

• 51337ms
• 48936ms
• 50563ms
• 48397ms
• 49962ms
• 50767ms
• 50829ms
• 48528ms
• 50715ms
• 50884ms

Stats (in ms):

• min: 48397
• max: 51337
• mean: 50091.8
• stddev: 1022.6636592741522

Notes:

• Majority of time is spent constructing stream, not streaming records

[nicholastmosher]

Closing this, since the scope of this was to measure baseline statistics. I'll link to this in followup issues focused on fixing delays where needed.