search

binarymatt / pyres

a resque clone in python
http://github.com/binarydud/pyres
MIT License
955 stars 130 forks source link

fork before processing jobs in the minion, just like the worker #97

Closed joeshaw closed 12 years ago

joeshaw commented 12 years ago

Allows the OS to reclaim resources used by jobs as soon as their done

joeshaw commented 12 years ago

hold off on this for a little bit; I'm not quite happy with how the result of 88f0d55 ends up in my log processing software. I'll submit an update for it shortly. The core of the patch is in 19939e1, though, and that's still sound.

joeshaw commented 12 years ago

ok, i am happier with this version.

binarymatt commented 12 years ago

originally the manager/minion model was created because forking on every job is costly in some situations. Wouldn't this cause the same problems?

If the minions are each forking on every job, why use them instead of normal workers

joeshaw commented 12 years ago

The main reason is from a monitoring/management point of view, it's easier to start up one process (the manager) and have it manage its subprocesses rather than starting a bunch of worker processes.

I wasn't aware that the original minion model was trying to avoid the fork overhead, that makes sense.

What do you think is the right thing to do here? Should I make it an option to pyres_manager? Should I change my workers so that they fork themselves? Should I just leave this in my own pyres fork?

binarymatt commented 12 years ago

i'm going to go ahead and close this ticket. I'm not sure how to keep things fast, but also fork the managed processes. Perhaps there is room for a worker pool class that manages workers. I'll have to put this in the issues list