search

LearnersGuild / echo

learning management system
MIT License
3 stars 31 forks source link

Move changefeed listeners to web service #1057

Open heyheyjp opened 7 years ago

heyheyjp commented 7 years ago

Currently, the workers set up listening to db changefeeds and, effectively, queue items for their own work queues. This is problematic because if changes to the database occur that should be handled by a worker while the worker is not running, that change event is lost and the job is never processed. Instead of connecting the changefeed listeners in the worker process, we should do it in the standing web service. This also makes it easier to choose to alternative mechanisms for background task execution (instead of having always-running worker dynos).

jeffreywescott commented 7 years ago

For what it's worth, these used to be in the web service, but it caused performance problems during "heavy" load (i.e., voting), since JavaScript is single-threaded.

heyheyjp commented 7 years ago

Ah. Good to know. We might find this interesting in thinking about how to attack the issue: https://www.rethinkdb.com/docs/sharding-and-replication/#running-a-proxy-node