binarymatt / pyres

a resque clone in python
http://github.com/binarydud/pyres
MIT License
955 stars 130 forks source link

Registers worker again after a Redis flush #121

Closed guilhermef closed 11 years ago

guilhermef commented 12 years ago

I was having problems after a flush on Redis. The worker was still running but It wasn't on the workers list. I thought it would be better to just register it again.

binarymatt commented 12 years ago

when you say flush, are you talking about running flushdb or flushall on the redis server?

guilhermef commented 12 years ago

It's flushall

binarymatt commented 12 years ago

if a flushall command is run while workers are attached to a redis server, then nothing is guaranteed to be around, including worker state, queues, stats, etc.

I like the is_registered function, but my concern is that you are introducing more traffic between the worker and the data server during the work cycle, which is supposed to be as fast as possible.

guilhermef commented 12 years ago

I could change the behavior to check it after 3 iterations instead of every time.

guilhermef commented 11 years ago

@binarydud did you thought about that ?

binarymatt commented 11 years ago

Sorry about not getting back to you quickly.

I've thought about it and I'm not sure that checking for registration in the middle of doing work is a valid use case. Generally it's not considered good practice to flush your entire database while work is being performed, so I'm not too keen to introduce code that is only useful in that case. What is the use case for flushing your entire db while workers are connected and working?

guilhermef commented 11 years ago

I had to do this because in my company when the redis server fails, the connection is redirected to an empty redis. So I have to register the worker again on this new one.

binarymatt commented 11 years ago

Ok, that's an interesting case and it made me think about another direction.

After reading the docs for redis again, what if instead of checking for registration, the workers always call the register method. That way instead of a possible 2 roundtrip db hit, it's always just a 1 roundtrip.

guilhermef commented 11 years ago

Yeah, that's probably the best option. I've already changed the pull request