jjbuchan
commented
5 years ago On the other hand maybe rebalancing is a premature optimization at this point, and a simple workq is sufficient.
Thinking about current services that don't rebalance and the problems it causes us:
- Noit
- Luckily doesn't break much, but we have had times where numerous collectors have been down for an extended period.
- Having to manually offload checks from down collectors is a pain
- Means checks won't run in that zone. At least means we're now doing a quorum of 2 zones; at worst means we're not monitoring them at all.
- AEPs / SNET Proxy
- We have to restart all of them at once to try get things balanced
- Trying to fix it triggers a large amount of unnecessary operations, as we have to break down "good" connections
- Very painful
- Having rebalancing built in should make life much easier for us.
I'd rather we consider it in the design early than try to force it in later.
itzg
GeorgeJahad
Atomix
First, I looked at the workq described here: https://atomix.io/docs/latest/api/io/atomix/core/workqueue/WorkQueue.html
While it does some of what the WorkAllocator does, it doesn't seem to address the cross worker rebalancing problem that WorkAllocator implements. If we want that, then we still have to implement much of what is in WorkAllocator. On the other hand maybe rebalancing is a premature optimization at this point, and a simple workq is sufficient.
However, Atomix workq also seems to require some cluster configuration, because the workq is implemented over a cluster created by the user nodes. Since we are already doing that for etcd, it seems a bit redundant.
Bottom line, Atomix is not a clear winner to me, (although it probably would be if it was built on top of etcd,) but I wouldn't object to using it if you others felt differently.
I've googled around some but haven't found a better etcd specific alternative to Atomix workq's.
WorkAllocator specific comments:
I'm confused by this is test: https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L404 We own the active key, so don't we already know it exists?
Just to be clear, version "0" here means it doesn't exist?: https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L488
If the transaction failed, but we stop processing here, then we still have the active key, so no one else is going to process either, right? That seems like a problem: https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L423 Similarily, this gets popped even if the txn fails: https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L369
Don't we have to guard these in a try/finally: https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L391 https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L474
we are starting from rev 0 here. will that lead to a lot of unneeded work? https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L252 https://github.com/itzg/try-etcd-work-allocator/blob/master/src/main/java/me/itzg/tryetcdworkpart/services/WorkAllocator.java#L330
introductory Workallocator documentation that would have helped me.
The code in this module allows processes to add/update/delete tasks to a distributed workq and allows the workers processing the q to be aware of each other and increase/reduce their share of the total work, as workers come and go.
data structures
The workq is implemented with a set of etcd kv's prefixed with the name "registry". Each new task in the q is a new kv pair with that prefix.
As workers pick up tasks from the q, they create an analagous entry in etcd with the prefix "active". That way the other workers know the task is in progress.
Finally, there is a set of kv pairs with the prefix "workers". This allows the workers to know of the existance of the others so they can shed work if the load becomes unbalanced.
threads
This module manages the work/worker q's by creating three threads, (each of which watches one of the prefixes listed above):
The registry thread watches for incoming work in the "registry" kv pairs and "grabs" it in a etcd transaction to ensure no two workers grab the same task.
The worker thread checks for incoming workers; as new workers arrive, it determines what should be it's new share of the total load and sheds tasks when it determines it has too many.
Finally, the active thread checks for tasks shed by other processes and attempts to grab them if the current process is underloaded.