69mb
commented
12 years ago Got any specific issues in mind since the framework as it is right now is already pretty extensible. New services only need to start consuming and possible publishing to any of the general purpose queues to get started. So here is my thinking behind the current framework:
Very soon we will need to start pre-calcuating data for the trends. This will involve a new application that attaches to the metadata exchange for droplets that have just come and that would be it; the app has all the data it needs to to proceed. The same would apply for logging, transforming droplet_content or any new service that needs to work on new droplets.
The topic queues on the chatter exchange was also to provide limitless possibilities for expansion. Currently its used by rss to listen for changes in rss channel options and update accordingly and the twitter consumer should do the same. Any other number of topics can be published to this exchanged and any manner of services can come up to consumer them.
All of this is done with applications that are autonomous as much as possible and do not rely on other apps in the ecosystem to provide service(otherwise we create bottlenecks) and scaling is simply a factor of increasing a server's size and the number of workers or adding new servers and workers.
ekala
Re-organise (refactor) the code service stack for easier extensibility. While at it also document the same.