search

bencheeorg / benchee

Easy and extensible benchmarking in Elixir providing you with lots of statistics!
MIT License
1.42k stars 66 forks source link

timeout error when running benchmarks #71

Closed devonestes closed 7 years ago

devonestes commented 7 years ago

Guten Tag!

It's probably not a big deal, but I was running some benchmarks just now, and on one run I got the following timeout error from Task.await.

$ mix run code/general/map_lookup_vs_pattern_matching.exs
Elixir 1.4.2
Erlang 19.2.3
Benchmark suite executing with the following configuration:
warmup: 2.0s
time: 10.0s
parallel: 1
inputs: none specified
Estimated total run time: 24.0s

Benchmarking Map Lookup...
Benchmarking Pattern Matching...
** (exit) exited in: Task.await(%Task{owner: #PID<0.70.0>, pid: #PID<0.174.0>, ref: #Reference<0.0.1.458>}, 5000)
    ** (EXIT) time out
    (elixir) lib/task.ex:416: Task.await/2
    lib/benchee/utility/map_value.ex:45: anonymous fn/1 in Benchee.Utility.MapValues.p_map_values/2
    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
    lib/benchee/utility/map_value.ex:44: Benchee.Utility.MapValues.p_map_values/2
    lib/benchee/statistics.ex:68: Benchee.Statistics.statistics/1
    lib/benchee.ex:36: Benchee.do_run/2

I ran the same task again right after and it worked just fine, so it's a transient issue. I figured I'd file the error just so you were aware, but clearly it's not really that much of an issue since it fixes itself. However, if there's anything else I can do to help shed some light on this, then let me know and I'll send along whatever I can!

PragTob commented 7 years ago

Thanks a ton!!!!!! :heart:

This is great, really keep it up even with little things. Otherwise I'd never know. These issues are super valuable as a maintainer, which is why I always wonder why not more people open a bunch of bugs in OSS projects because there's definitely a tone of them.

This is a new usage of Task.await introduced for calculating statistics in parallel. I suppose you do rather micro benchmarks for statistics calculations to take more than 5 seconds?

Will set it to infinity as well. :)

Also while not grave, certainly annoying. I might cut a new release sooner than I originally planned to :D

PragTob commented 7 years ago

Fixed on master, although I'm not sure you wanna use master yet as it breaks the other formatters (HTML etc.) right now as one main key was renamed - shouldn't be too long though ;P

devonestes commented 7 years ago

Awesome - happy to help!

Yeah, I was benchmarking some pretty fast stuff. Here was the second successful run for your reference:

$ mix run code/general/map_lookup_vs_pattern_matching.exs
Elixir 1.4.2
Erlang 19.2.3
Benchmark suite executing with the following configuration:
warmup: 2.0s
time: 10.0s
parallel: 1
inputs: none specified
Estimated total run time: 24.0s

Benchmarking Map Lookup...
Benchmarking Pattern Matching...

Name                       ips        average  deviation         median
Pattern Matching      726.52 K        1.38 μs   ±376.42%        1.20 μs
Map Lookup            455.62 K        2.19 μs   ±185.63%        1.90 μs

Comparison:
Pattern Matching      726.52 K
Map Lookup            455.62 K - 1.59x slower

Thanks again for the awesome library! I look forward to helping more in a week or so once I get my open source time back 😄

PragTob commented 7 years ago

Cool stuff! But damn that standard deviation... Wish I had implemented #50 again so the console formatter could show min/max and sample size. Or the nth percentile thing. So much to do :D

Your talk is next week, right? Good luck + much success!

If you do more complex stuff in the benchmarking functions, #70 might be interesting to you. If you try it out, the difference would be interesting for me to see :)