search

nextml / NEXT

NEXT is a machine learning system that runs in the cloud and makes it easy to develop, evaluate, and apply active learning in the real-world. Ask better questions. Get better results. Faster. Automated.
http://nextml.org
Apache License 2.0
160 stars 54 forks source link

fixed bug with caching wrong minionworker hostname #198

Closed dconathan closed 7 years ago

dconathan commented 7 years ago

Fixes #193

lalitkumarj commented 7 years ago

I merged this in. Can you say a few more words about what the issue was?

On Fri, Jun 30, 2017 at 4:56 PM, dconathan notifications@github.com wrote:

Fixes #193 https://github.com/nextml/NEXT/issues/193

You can view, comment on, or merge this pull request online at:

https://github.com/nextml/NEXT/pull/198 Commit Summary

  • fixed bug with caching wrong minionworker hostname

File Changes

Patch Links:

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nextml/NEXT/pull/198, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWhGKLnhsSvCw9ZeEqCzShvFYim3IAyks5sJWEUgaJpZM4OLCzX .

dconathan commented 7 years ago

Basically, the broker was caching the hostname (which is the same as the docker id) of the minion_worker container in the rabbitmqredis container. The rabbitmqredis redis db would persist even if you shut down the backend. If the backend was shut down long enough (or maybe whenever you restarted your computer?), docker would assign minion_worker a new hostname/id. The broker would keep trying to hit the old hostname from cache and nothing would happen.

You could see this happening in real time because the terminal would print "Hostname \<whatever> from Redis" - and it'd be the wrong hostname.

New code: minion_worker hostname/id only persists in redis for 10 minutes, and introduced another method for getting the hostname for when the minion_worker itself is asking for its own hostname.

lalitkumarj commented 7 years ago

I wonder if this was behind a lot of the bugs we were having with 'workers dropping'. Good catch!

On Mon, Jul 10, 2017 at 2:41 PM, dconathan notifications@github.com wrote:

Basically, the broker was caching the hostname (which is the same as the docker id) of the minion_worker container in the rabbitmqredis container. The rabbitmqredis redis db would persist even if you shut down the backend. If the backend was shut down long enough (or maybe whenever you restarted your computer?), docker would assign minion_worker a new hostname/id. The broker would keep trying to hit the old hostname from cache and nothing would happen.

You could see this happening in real time because the terminal would print "Hostname from Redis" - and it'd be the wrong hostname.

New code: minion_worker hostname/id only persists in redis for 10 minutes, and introduced another method for getting the hostname for when the minion_worker itself is asking for its own hostname.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nextml/NEXT/pull/198#issuecomment-314197286, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWhGAor0x1Sb3fU74gEqjsSbeDbOP9Mks5sMnBAgaJpZM4OLCzX .