Iterations per second appears to be for one process, not aggregate

[scouten]

See discussion in https://elixirforum.com/t/looking-for-help-with-poor-ecto-query-performance/25476/13:

I would have expected the sleep process in this example to have completed ~10 iterations per second when 10 parallel processes are running. Instead benchee is reporting only 1.

[OvermindDL1]

To be clear, @scouten is referencing this:

Benchee.run(
  %{
    "sleep" => fn ->
      Process.sleep(1_000)
    end
  },
  parallel: 10,
  time: 10
)

warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 10
inputs: none specified
Estimated total run time: 12 s

Benchmarking sleep...

Name            ips        average  deviation         median         99th %
sleep          1.00         1.00 s     ±0.02%         1.00 s         1.00 s

Which seems entirely accurate to me. If it reported an IPS of 10 that would seem absolutely outright wrong.

Thus I'm unsure of the reason for this issue, other than that perhaps on the parallel wiki page it should mention that the iteration count is per benchmark scaffold, not for the parallel collection of them.

[PragTob]

Hi there @scouten :wave:

Thanks for the report. This is entirely the intended behavior. @OvermindDL1 is completely right I'll try to improve the wiki page and make it more clear.

I like the feature (partially because it was a small change that made someone be able to use benchee for their purposes of more load testing) but it has been misunderstood/is kinda difficult to explain.