Open ioquatix opened 7 years ago
I think I found the cause of the problem. There is an excessive amount of logging due to $DEBUG being set in Travis. For these performance specs, perhaps it should be disabled.
Travis is generally an unreliable place to run benchmarks
I appreciate that feedback.
It's a relative benchmark so it's comparing two things. It's not the absolute performance, but the relative performance that indicates a regression and it's a useful check.
I've been running these kinds of performance checks for a long time. While I don't expect travis to be fast, I've never had any specific issue with their performance.
The problem here is the use of $DEBUG to determine a default log level here:
https://github.com/socketry/async/blob/master/lib/async/logger.rb
I think it's not a bad choice except it's causing issues in this particular case. The solution is to disable it for the performance specs e.g. before(:all) { Async.logger.level = Logger::WARN }
and the appropriate one afterwards, but there is more than one performance spec, so a shared context would be good here. I actually have a shared context in another project which includes support for Benchmark
, RubyProf
and some other basic profiling tools, it's really easy to use. I'll probably copy it across.
We should be both faster than Bind9 and native resolver. Check why this has regressed.