Send events to the batch endpoint instead of the single event endpoint

[martin308]

Changes the endpoint that we send events to the batch endpoint.

The previous flow of events was like this:

• event
• queue
• threads * max_concurrent_batches
  • send event to events endpoint and push response to responses queue

we now have the following flow:

• event
• queue
• thread
  • read event off queue
  • add to hash keyed on dataset, apikey, and apihost
  • every send_frequency
  • take the hash and split each set of events into groups of max_batch_size
  • push each batch onto queue
• threads * max_concurrent_batches
  • serialise each event to JSON, send errors to responses queue
  • send each serialised batch to batch endpoint
  • send errors to the responses queue
  • read response and push a response onto the responses queue for each event status returned in the response

Benchmarking

Using the following code, to send 1000 events using both the current version of libhoney and this new method

require "benchmark"
require "libhoney"

libhoney = Libhoney::Client.new(writekey: "...", dataset: "libhoney-benchmark")

b = Benchmark.measure do
  1000.times do
    event = libhoney.event
    event.add_field("look", "here")
    event.send
  end

  libhoney.close(true)
end

puts b

Current version:

user	system	total	real
2.020500	0.119861	2.140361	( 7.310318)
1.965627	0.117135	2.082762	( 7.354831)
2.012084	0.123885	2.135969	( 7.436962)

New version:

user	system	total	real
0.107759	0.041983	0.149742	( 0.493777)
0.091437	0.039685	0.131122	( 0.530462)
0.111085	0.037703	0.148788	( 0.482274)

[christineyen]

removing myself as a reviewer but leaving EXTREMELY POSITIVE FEELINGS in the air ✨