Add Benchee benchmarks for Plugs

[GregMefford]

This is an initial stab at adding Benchee benchmarks for the Spandex.Plug.* modules to get a feel for how much performance impact there is when adding Spandex to a service. It is important to note that since the BaselineRouter doesn't really do anything, the x slower metrics are meaningless, so we should focus on the raw per-request timing increases due to adding tracing (in terms of how many microseconds is tracing adding to each request we process).

I wanted to get this in place so that I can work on an implementation for trace sampling (hence the stubbed-in inputs list) and validate that it alleviates the performance overhead of tracing, at least on average, when only sampling a small fraction of requests.

Feedback welcome!

Some initial results from my laptop (with a million other things running, so take it with a grain of salt):

Operating System: macOS"
CPU Information: Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.6.6
Erlang 20.3.8.1

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 4
inputs: no_sampling
Estimated total run time: 42 s

Benchmarking addcontext with input no_sampling...
Benchmarking baseline with input no_sampling...
Benchmarking start_finish with input no_sampling...

##### With input no_sampling #####
Name                   ips        average  deviation         median         99th %
baseline           49.04 K       20.39 μs    ±82.41%          18 μs          44 μs
start_finish        5.62 K      178.03 μs    ±45.97%         168 μs         427 μs
addcontext          4.74 K      210.96 μs    ±73.94%         185 μs      771.44 μs

Comparison:
baseline           49.04 K
start_finish        5.62 K - 8.73x slower
addcontext          4.74 K - 10.35x slower

Memory usage statistics:

Name                 average  deviation         median         99th %
baseline             3.81 KB     ±0.00%        3.81 KB        3.81 KB
start_finish        24.76 KB     ±0.02%       24.77 KB       24.77 KB
addcontext          33.67 KB     ±0.02%       33.67 KB       33.67 KB

Comparison:
baseline             3.81 KB
start_finish        24.77 KB - 6.50x memory usage
addcontext          33.67 KB - 8.83x memory usage

[coveralls]

Coverage remained the same at 70.647% when pulling be38f03cd4a8d433a858e67e41895f276859cb8e on GregMefford:benchee into 4028873ff64d6ac9b15804a466ba96cf24df8f3c on zachdaniel:master.

[zachdaniel]

This looks good to me. Let me know when you are ready and I will merge.

[GregMefford]

I think this is ready to merge when you are. I'll send the sampling thing as a separate PR when I get it figured out. The only issue with this one is that it's currently tangled with the Datadog sender API because it wasn't obvious to be how to use the test ones you made, and actually it kind of makes sense to test the sender when doing a benchmark since it's a thing that happens in production use-cases. Maybe this benchmark should only test the Spandex code, though, and we would separately benchmark the SpandexDatadog code? I'll need to think more about that, but I think this PR is fine for now.

[zachdaniel]

I think I might want to merge the adapter split first, and then merge this. There is a test adapter that is platform agnostic that could be used for these benchmarks in that new branch.

[GregMefford]

Yep, I agree that’s a better choice.

On Fri, Aug 17, 2018 at 6:02 PM Zach Daniel notifications@github.com wrote:

I think I might want to merge the adapter split first, and then merge this. There is a test adapter that is platform agnostic that could be used for these benchmarks in that new branch.

— You are receiving this because you authored the thread.

Reply to this email directly, view it on GitHub https://github.com/zachdaniel/spandex/pull/59#issuecomment-413999104, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEPW14O9VDyvz_e9rHzWdHhW8rN15Z_ks5uRz1egaJpZM4WAgoi .

[zachdaniel]

Alright, the other branch is merged. Not going to cut a release just yet though, want to let it mellow and test it out a bit.

[GregMefford]

Cool. I will work on rebasing this to use the test adapter.

On Fri, Aug 17, 2018 at 10:51 PM Zach Daniel notifications@github.com wrote:

Alright, the other branch is merged. Not going to cut a release just yet though, want to let it mellow and test it out a bit.

— You are receiving this because you authored the thread.

Reply to this email directly, view it on GitHub https://github.com/zachdaniel/spandex/pull/59#issuecomment-414027490, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEPWwo5u8iT5ey0iIW7YhaiO3BzCYO-ks5uR4E8gaJpZM4WAgoi .

[sourcelevel-bot[bot]]

Ebert has finished reviewing this Pull Request and has found:

• 1 possible new issue (including those that may have been commented here).
• 1 fixed issue! 🎉

You can see more details about this review at https://ebertapp.io/github/zachdaniel/spandex/pulls/59.

[GregMefford]

OK, I got this rebased to remove the dependency on DataDog - it's just benchmarking performance of the code Spandex library now.

Should be ready to merge now if you're happy with it. ✅

[zachdaniel]

Awesome! Merged :)