search

fanout / django-eventstream

Server-Sent Events for Django
MIT License
638 stars 84 forks source link

Updating EventCounter takes a comparatively long time #103

Open Schlepptop opened 1 year ago

Schlepptop commented 1 year ago

I am trying to improve the performance of my application, and I noticed that updating the eventcounter (which is done for every sent event) performs 3 DB Queries, which on my machine in total take about 7ms. I am sending lots of events, often to many channels, so these quickly add up. Is there anything I can do to reduce the impact of updating the EventCounter?

jkarneges commented 1 year ago

The built-in storage implementation (DjangoModelStorage) writes events to the database and may not be optimized for lots of events. Appending takes a row lock on the EventCounter model which hopefully doesn't block other rows, but I have not benchmarked if that's really true and it may depend on the database engine.

You have three options:

a) Try to optimize the current code and submit a PR. Maybe the transaction isn't written optimally. b) Write your own StorageBase implementation. This would give you the most control for high scale, to do things like sharding or using non-SQL data sources. c) Don't use event storage and treat event delivery as best-effort.

Schlepptop commented 1 year ago

Thank you for your reply! This doesn't help anyone else, but I have decided to solve this and other bottlenecks I faced by rewriting my high frequency async communication backend in golang. Looking at the EventCounter model implementation, I noticed that it selects the value, increments it, and then updates, instead of doing one relative Update with an F() field, but I'm afraid I don't have an idea how to improve that if we need the value to create the zero_event if necessary.