Add benchmarks

[codefromthecrypt]

From @adriancole on November 17, 2015 3:21

Some aspects of code need to be benchmarked as they will be run a lot.

Here's some ideas... feel free to contribute!

For each scenario, use traces of 1-5 depth and 10-200 width, where spans range from local to remote w/up to 2KB annotation sizes.

Library Microbenchmarks:

• thrift and json codec
• time skew adjustment

Integration Benchmarks (ex server running against each spanstore impl)

• accepts/second for 1, 10, 100 length spans in json vs thrift
• at rest size of 10K - 1M traces:
• service/span name query latency
• duration query latency
• dependency query latency

Copied from original issue: openzipkin/zipkin-java#45

[codefromthecrypt]

55 is a start, benchmarking the sampling algo

[codefromthecrypt]

serialization benchmarks added https://github.com/openzipkin/zipkin-java/pull/173

[codefromthecrypt]

related work:

zipkin-reporter has a micro(ish)benchmark which tests the throughput of senders like Kafka (not directly to a zipkin server, but could be rejigged to)

Ex. https://github.com/openzipkin/zipkin-reporter-java/blob/master/benchmarks/src/main/java/zipkin/reporter/HttpSenderBenchmarks.java

The sender itself is used by many instrumentation projects (like brave, spring etc), but it was also made to facilitate benchmarking a real setup. The idea was to use something like jbender and hook it up to a reporter and a generator. It would need a plugin to monitor backup etc.

There was also an idea to use spigo to simulate loads. This stopped working at some point and also needs help to route through our go library as opposed to creating dump files.

Another idea was to use storage-specific benchmarks, such as Rally for elasticsearch #1354 or stress tool for cassandra. While not flow tests, these could help isolate storage performance concerns (like indexing)

We've also had a number of ad-hoc tests where people make python scripts to test kafka. Here's an example of such a script: https://github.com/openzipkin/zipkin/issues/1141#issuecomment-228514983

Another way you could test is shoving traffic through an existing system as a black box. For example, you could look at polyglot-zipkin and drive traffic through that (via grinder or similar) and watch side-effects on the storage system or collector metrics of zipkin

[codefromthecrypt]

used a quick and dirty wrk benchmark to measure overhead reduction of tomcat -> undertow https://github.com/openzipkin/zipkin/pull/1806

[codefromthecrypt]

here's a lua file for wrk, though it is a bit crap as it uses the same post every time (and the trace ID gen is janky)

-- crappy trace id generator
local random = math.random
local trace_template ='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
local trace_id = string.gsub(trace_template, '[xy]', function (c)
  local v = random(0, 0xf) or random(8, 0xb)
  return string.format('%x', v)
end)

-- ex luarocks-5.1 install luaposix
local P = require "posix"
local tim = P.gettimeofday()
local timestamp = tim.sec..tim.usec
local timestamp_foo = tim.sec..(tim.usec + 2500)
local timestamp_bar = tim.sec..(tim.usec + 10500)

wrk.method = "POST"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Origin"] = "http://foobar.com"
wrk.body   = '[{"traceId":"'..trace_id..'","id":"4d1e00c0db9010db","kind":"CLIENT","name":"get","timestamp":'..timestamp..',"duration":207000,"localEndpoint":{"serviceName":"frontend","ipv6":"7::0.128.128.127"},"remoteEndpoint":{"serviceName":"backend","ipv4":"192.168.99.101","port":9000},"annotations":[{"timestamp":'..timestamp_foo..',"value":"foo"},{"timestamp":'..timestamp_bar..',"value":"bar"}],"tags":{"http.path":"/api","clnt/finagle.version":"6.45.0"}}]'