Resilient verk improvements

edgurgel / verk

A job processing system that just verks! 🧛‍

https://hex.pm/packages/verk

MIT License

721 stars 65 forks source link

Resilient verk improvements #180

Closed edgurgel closed 5 years ago

edgurgel commented 5 years ago

Here are some changes I made from the original PR #159

Change so that QueueManager maintains verk_nodes and verk:node:#{node_id}:queues up-to-date. It avoids possible de-synchronization between Redis and local state of the Verk node. The main idea behind this changes is: "If any job is added to the inprogress list, this node and this queue must be tracked so that other nodes can rescue their failure if it happens." QueueManager will conditionally maintain these data structures if generate_node_id is true.
Change Node.Manager to not crash if heartbeat failed.

coveralls commented 5 years ago

Coverage decreased (-0.5%) to 88.679% when pulling 0bb216447566566e8bd9e91f710a2fedbde8b704 on resilient-verk-improvements into 36666a97d8f8455b4e83560f9a1cdfe868b4d8e9 on master.

coveralls commented 5 years ago

Coverage decreased (-0.5%) to 88.679% when pulling 0bb216447566566e8bd9e91f710a2fedbde8b704 on resilient-verk-improvements into 36666a97d8f8455b4e83560f9a1cdfe868b4d8e9 on master.

coveralls commented 5 years ago

Coverage decreased (-0.5%) to 88.679% when pulling 0bb216447566566e8bd9e91f710a2fedbde8b704 on resilient-verk-improvements into 36666a97d8f8455b4e83560f9a1cdfe868b4d8e9 on master.

coveralls commented 5 years ago

Coverage decreased (-0.5%) to 88.679% when pulling be7d7693beec411c120ef39a9633ce19c3ad40e1 on resilient-verk-improvements into 36666a97d8f8455b4e83560f9a1cdfe868b4d8e9 on master.

edgurgel commented 5 years ago

Updated algorithm:

Each time a job is moved to the list of jobs inprogress of a queue this node is added to verk_nodes (SADD verk_nodes node_id) and the queue is added to verk:node:#{node_id}:queues (SADD verk:node:123:queues queue_name)
Each frequency seconds we set the node key to expire in 2 * frequency PSETEX verk:node:#{node_id} 2 * frequency alive]
Check for all the keys of all nodes. If the key expired it means that this node is dead.
To restore we go through all the running queues (verk:node:#{node_id}:queues) of that node and enqueue them from inprogress back to the queue. Each "enqueue back from in progress" is atomic (<3 lua) so we won't have duplicates.

How to use:

Set the application env generate_node_id to true

If it's not true it won't use this new code. It will basically work as before.