Disabled tracepoints introduce a performance degradation

[vladzcloudius]

Two tracepoints from "tcp_output.cc: added tracepoints allowing to debug the Tx flow" patch (not yet committed): trace_tcp_output() and trace_tcp_output_just_ret() are responsible for 1.39% degradation in a 64B netperf test on muninn-huginn setup.

When we compile out the tracepoints completely it delivers 3.25% compared to the clean master (973107d13) and 4.7% compared to the same clean master with the "tcp_output.cc: added tracepoints allowing to debug the Tx flow" patch.

How to reproduce:

• checkout OSv with the following 973107d13 hash.
• Configure the remote host (huginn):
  • Fedora20 with 3.14.9-200.fc20.x86_64 kernel.
  • Start a netserver and configure its affinity to CPUs 0 and 1: sudo taskset -pc 0,1 pgrep netserver.
  • Configure NIC's IRQs affinities: ./set_affinity_qemu.sh p4p2
• Start the netperf test on muninn: sudo scripts/run.py -m8G -c1 -nv -b bridge2 -e "/tools/netperf.so -H 192.168.2.2 -l 30 -- -m 64
• Configure the qemu, vhost and NIC's IRQs affinities on muninn: ./set_affinity_qemu.sh p4p2
• The set_affinity_qemu.sh test may be found at https://github.com/vladzcloudius/scripts.git
• The results are as follows (I ran 10 interations for 30 seconds each):
• Clean 973107d13: 1770.89 Gbps, COVAR: 0.24%
• 973107d13 with "tcp_output.cc: added tracepoints allowing to debug the Tx flow": 1746.35 Gbps, COVAR: 0.4%
• Clean 973107d13 with traces compiled out: 1828.47 Gbps, COVAR: 0.19%

[glommer]

Okay, I've misheard you when you explained this through google hangout, and heard a much higher figure.

Those numbers are actually consistent with what I had already found - and reported -for redis.

On Thu, Jul 31, 2014 at 5:57 PM, vladzcloudius notifications@github.com wrote:

Two tracepoints from "tcp_output.cc: added tracepoints allowing to debug the Tx flow" patch (not yet committed): trace_tcp_output() and trace_tcp_output_just_ret() are responsible for 1.39% degradation in a 64B netperf test on muninn-huginn setup.

When we compile out the tracepoints completely it delivers 3.25% compared to the clean master (973107d https://github.com/cloudius-systems/osv/commit/973107d13661b0001311c5015f586768453c815e) and 4.7% compared to the same clean master with the "tcp_output.cc: added tracepoints allowing to debug the Tx flow" patch.

— Reply to this email directly or view it on GitHub https://github.com/cloudius-systems/osv/issues/428.

[wkozaczuk]

@glommer Shall we close it?

[nyh]

I don't know, may we should leave it as a suggestion that tracepoints aren't as free (or nearly free) as we thought, and maybe we should rethink their implementation the future? Although, to be honest, I doubt we will. They are to convenient to give up, and it won't be easy to come up with an even lower-overhead implementaiton.