search

tj / axon

message-oriented socket library for node.js heavily inspired by zeromq
MIT License
1.5k stars 155 forks source link

round robin in req / resp socket #103

Closed ebensing closed 11 years ago

ebensing commented 11 years ago

I am building a system that is essentially distributing a bunch of processing tasks across a large number of nodes.

I am trying to distribute these evenly across the nodes such that once a node finishes its work, it will receive the next task.

So my general question is: are nodes that have not responded to the previous request considered in the round robin queue? and if they are, would implementing a mode to allow for this be difficult? (I'm happy to implement it, just wanted to check in)

ebensing commented 11 years ago

Got a chance to glance around at the code. Judging by this line it seems it does not remove nodes from the pool until a response is heard. Would you have any interest in a pull request to add such functionality as an option?

tj commented 11 years ago

nah because you could quickly exhaust workers and have nothing to send to, too arbitrary for core I think

ebensing commented 11 years ago

Would a mode that sends to the node with the least number of outstanding requests work then? so, you wouldn't run into the case where you have nothing to send to. Just need to do a little extra bookkeeping.

tj commented 11 years ago

yeah that would work, more generic at least

ebensing commented 11 years ago

Awesome. I'll get to work on that then and shoot a pull request over.

gjohnson commented 11 years ago

That was kinda the plan for the "plugins". However, they will need some tweaking to support this level of flexibility though. Would be great if you could incorporate them in what you do.

tj commented 11 years ago

@gjohnson yeah I still like that approach too, +1 from me

ebensing commented 11 years ago

@gjohnson @visionmedia By plugins, do you mean using the plugin method on the sockets that you have marked as private? I had originally started building this using that API and essentially using it to monkey patch the necessary methods.

I'll try thinking about the plugin system a bit and see if there would be any way to design a decent API for it that would allow for a plugin like this. Can't make any promises though, currently a little strapped for time.

ebensing commented 11 years ago

Just sent over the PR. I investigated trying to expand the plugin system to allow for this, but I couldn't come up with an API I liked that would expose the necessary functionality for something like this.