snowplow-incubator / common-streams

Other
1 stars 0 forks source link

Reduce thread context switching in pubsub source #36

Closed istreeter closed 11 months ago

istreeter commented 1 year ago

Previously we were running dispatcher.unsafeRunsync(io) for every single message we received from pubsub. This is a problem, because the io gets executed on the cats-effect io compute pool. So in order to just receive a message, the app needs to context switch from the subscriber thread to the cats-effect thread and back again. This is very inefficient, especially as we do it for every single event.

The solution is to stop using a Dispatcher inside the subscriber callback.