The solution implemented for #133 is suboptimal in two aspects:
It uses the redis KEYS command, which can be bad for performance.
It does something very similar to the heartbeat checker, and I think it should be possible to modify the heartbeat checker to make the stale key checker obsolete. This would leave us with less code, less threads, and easier maintenance.
The solution to 1. is rather straightforward: KEYS is used to find all workers that have their activeKey set. We could either
Add a redis set that keeps track of all activeKeys that are set. A worker would add its WorkerId to this set when it takes a job, and it would be removed when the activeKey is cleared.
Move away from using a separate activeKey for each worker, and store the map WorkerId -> RequestId in a redis HASH instead.
I'm leaning towards the latter option.
I think 2. should be possible by changing the heartbeat checker to check the heartbeats of all workers that satisfy at least one of the following
they appear as a live worker in activeOrUnhandledWorkers
they have their activeKey set (or occur in the HASH suggested in the solution to 1. above).
The solution implemented for #133 is suboptimal in two aspects:
KEYScommand, which can be bad for performance.The solution to 1. is rather straightforward:
KEYSis used to find all workers that have theiractiveKeyset. We could eithersetthat keeps track of allactiveKeys that are set. A worker would add itsWorkerIdto this set when it takes a job, and it would be removed when theactiveKeyis cleared.activeKeyfor each worker, and store the mapWorkerId -> RequestIdin a redisHASHinstead.I'm leaning towards the latter option.
I think 2. should be possible by changing the heartbeat checker to check the heartbeats of all workers that satisfy at least one of the following
activeOrUnhandledWorkersactiveKeyset (or occur in theHASHsuggested in the solution to 1. above).