How to measure performance of single emission?

[StreetStrider]

Hi, I'm working on measuring FRP libs performance. It is hard to do due to architectural differences, but I'm doing my best. I'm trying to measure a single tick or emit inside a single chain. I want libraries to have their preparation optimizations be taken into account. My first test looked like this:

        var n = 1

        var A = at(1, 17)

        var b = map(a => a + 1, A)
        var c = map(b => b + 1, b)
        var d = combine((a, c) => a + c + 1, A, c)

        var run = tap(() => n++, d)

        return () =>
        {
            return runEffects(run, newDefaultScheduler())
        }

Where outer code is a preparation code and inner function is a benchmarking function. This is working, but I think this internal function has all preparation code in runEffects, and I wanted preparation code to be outside. (I've noticed because this particular setup is quite slow)

My second approach was using adapter:

        var rs

        var n = 1

        var [ emit, A ] = createAdapter()
        // var A = at(1, 17)

        var b = map(a => a + 1, A)
        var c = map(b => b + 1, b)
        var d = combine((a, c) => a + c + 1, A, c)

        var run = tap(() => { n = n + 1; rs() }, d)
        runEffects(run, newDefaultScheduler())

        return () =>
        {
            var p = new Promise(rs_new => { rs = rs_new })
            emit(17)
            return p
            // runEffects(run, newDefaultScheduler())
        }

I'm not sure about the correctness of this approach, so I would like to get help from experts. So, to sum up, my question is how to properly test one emission. Any additional info is welcome as well.

[TylorS]

Hey @StreetStrider, your first approach looks correct,runEffects is effectful so it is already running by the time you attempt to await the promise it returns. There are some small tweaks that could be done to minimize the overhead of tasks, I threw together an attempt to minimize the overhead of such a test https://codesandbox.io/s/exciting-khayyam-bc4n4?file=/src/index.ts

[StreetStrider]

@TylorS thanks for the response. I think I need to clarify some parts after looking to your code. I saw you set a limiting via e = take(1, d). Indeed I try to benchmark single emission, but in reality benchmarking framework would run test function multiple times. That's why I'm taking an efforts to create setup function and offload everything except emission to it.

But I'm not familiar enough with most.js architecture. I believe this library features optimizations like fusion of consecutive maps (that's where I expect the maximum power of it), but I don't know where the optimizations live. Maybe they set up when I create connections with map and combine, or maybe when I run effects or when sinks are attached, I don't know. I want all preparations and optimizations to be outside, so I've tried to create single scheduler and runEffects once (in my second example).

I see it as a long-living event graph created once (benefitting from most.js optimizations + jit) and then benchmarking runs a single emission and make a mark when the data reaches effect (tap). The biggest problem I had here is that most.js is pull, so to force emission I ended up using adapter (not sure if it is correct).

The first example seemes correct, but I think all heavy logic would run on every benchmarking tick.

If I understood correctly, Sink in your example is equivalent to tap in mine. newDefaultScheduler + run must be equivalent to runEffects.

[TylorS]

The fusion optimizations all occur during the construction of the streams, prior to runEffects. When you call map/tap it constructs a class that will check to see if the previous Stream in the graph is of the same class and then will attempt to fuse them using function composition - see https://github.com/mostjs/core/blob/master/packages/core/src/fusion/Map.ts#L32. There are a few more optimizations like this I can reference if it helps, but I'd need to dig through the codebase again 😅

runEffects constructs a special Sink instance that is capable of disposing of the Stream's resources when it either fails or ends. Stream.run(sink, scheduler) returns a Disposable{ dispose: () => void } interface, which is used internally for the stream graph to stay resource-safe by combining Disposables. runEffects' Sink implementation will call disposable.dispose() for you when Sink.error or Sink.end are called utilizing Promise.reject and Promise.resolve respectively to complete the returned promise.

[StreetStrider]

@TylorS thanks for the sharing, this helped me a lot. Btw, moving from example 1 to example 2 on one of my cases changes numbers in such manner:

  diamond (most):
    812 ops/s, ±0.61% 

  diamond (most):
    617 500 ops/s, ±0.63%

This is mostly due to the running effects once for the whole case. Also, it is nice to know that fusion happens on constructing.

(I will close the ticket in a while)

[TylorS]

Feel free to reach out at any point!

[Frikki]

It’s been two weeks, @StreetStrider. Can we close the issue? Or do would you like to keep it open a bit longer?