Investigate getting current time from redis

[kantp]

We're having issues where clock skew adversely affects our heartbeat check.

We should consider using the redis TIME command http://redis.io/commands/time, instead of getUTCTime, so that we have one clock that all nodes agree on.

[kantp]

Using the current time from redis will involve an additional roundtrip to the database. @bitonic hat the idea to avoid that via lua scripting, but apparently using TIME in scripts is forbidden in redis, because it could lead to issues on a cluster.

I have another idea, that avoids the clock skew problems, and should also lead to simpler, and more robust, code in the heartbeat checker.

Instead of periodically updating a timestamp as its heartbeat, each worker will

• register itself once in a list of live workers
• periodically set a key with a timeout, http://redis.io/commands/expire. This key is the worker's heartbeat, and the time between updates should be shorter than the timeout.

The heartbeat check will

• for each of the live workers, check if the heartbeat is still in the database. If not, it is removed from the live worker list, and added to a list of dead workers. Jobs are re-scheduled.
• for each dead worker, check if the heartbeat failure was only temporary, by checking its heartbeat. Workers that resumed their heartbeat are added back to the list of live workers.

On the side of checkHeartbeats, we will still need a clock, but only for

• delaying checks
• registering the time at which a worker failed its heartbeat check.

For both purposes, a clock skew is not dangerous: at worst, we make more heartbeat checks than we had planned.

I'm currently exploring the implementation of this design.

[kantp]

I have made a slight modification to the design above

• the heartbeat checker does not check for workers that failed temporarily
• instead, workers do this themselves while sending heartbeats:
  • they try to update their last heartbeat
  • if they have not yet sent a heartbeat, or if the last heartbeat has already expired, this will fail, and the worker will add itself to the set of live workers.