I'm using criterion for measuring async-nats client performance.
As nats is a messaging system, to have meaningful results, each iteration needs to operate on thousands or millions of operations to yield meaningful results, not impacted by some internal API operations done before/afterwards.
That means, that each iteration can take up to ~15-30 seconds. Some API's also create short-living state on the server.
The problem is that warmup seems to not play nicely with those longer running state-creating iterations and pollute the results of test runs run after them (probably by abruptly shutting down long-running warmups that do not allow clearing the state?)
Would it be possible to optionally disable warmup?
I'm aware that that's proposing a solution, while I suspect the problem is that if warmup takes too long, it's shut down, but to be fair, in case of NATS warmup can anyway have some other negative side effects on bench that decrease the confidence in results, not raise it.
I'm using
criterionfor measuringasync-natsclient performance. Asnatsis a messaging system, to have meaningful results, each iteration needs to operate on thousands or millions of operations to yield meaningful results, not impacted by some internal API operations done before/afterwards.That means, that each iteration can take up to ~15-30 seconds. Some API's also create short-living state on the server.
The problem is that warmup seems to not play nicely with those longer running state-creating iterations and pollute the results of test runs run after them (probably by abruptly shutting down long-running warmups that do not allow clearing the state?)
Would it be possible to optionally disable
warmup? I'm aware that that's proposing a solution, while I suspect the problem is that if warmup takes too long, it's shut down, but to be fair, in case of NATS warmup can anyway have some other negative side effects on bench that decrease the confidence in results, not raise it.