honeycombio / libhoney-rb

Ruby library for sending data to Honeycomb
Apache License 2.0
11 stars 30 forks source link

Reduce Lock Contention #47

Closed sstelfox closed 3 years ago

sstelfox commented 4 years ago

While attempting to test honeycomb in our application we experienced a significant throughput performance decrease. While attempting to identify the root cause I noticed the potential for lock contention during this send_event.

Since the lock is only necessary if the client needs to be initialized, this simply moves the short circuit to avoid using the lock when it's not needed.

sstelfox commented 4 years ago

We're still in the process of tracking down what about adding the honeycomb client impacted our performance so much, but I'll get back to you when I learn more.

You're right this does change the semantics and does likely introduce a situation where two threads both don't see a client at the same time and one of them is replaced (though the creation of both with will still be thread synchronized), the last one will just win.

If the lock isn't necessary at all though, I would absolutely get rid of it here or at least move the initialization of the client somewhere that happens once so this much hotter path doesn't have to deal with the initialization and lock (which may just be a better solution if there is a better place to handle the initialization).

Locks are relatively expensive and the way it currently is, that lock penalty is being incurred for every call to this method. Moving the check outside the block also prevents the instantiation of unused objects during every call (though the lock performance likely outstrips any impact compared to those).

martin308 commented 4 years ago

I did some git spelunking to see if I could see if the lock was added for any specific reason. Unfortunately it's just under one large "Initial commit". I'll ask around internally to see if anyone knows. If not I say we just rely on ||= as there is no need for the lock

martin308 commented 4 years ago

@sstelfox after consulting with some internal folks about why this was done originally, I think the best course of action is to construct the transmission client in the initialize method and ditch the lock altogether. Let me know if you want to take this on or I can add it to my stack

robbkidd commented 3 years ago

We’re going to close this for now from inactivity. Feel free to re-open if the changes here are still relevant.