richardartoul / nola

MIT License
74 stars 6 forks source link

Huge Refactor (leader registry, memory-based load balancing, etc) #67

Closed richardartoul closed 1 year ago

richardartoul commented 1 year ago

This is a huge refactor to NOLA that does the following:

  1. WASM module registry is now separate from the registry itself. This narrows the scope of what methods the registry needs to implement.
  2. Removed support for built in KV functionality within the registry itself. This makes implementing new registries easier and KV can always be injected into actor implementations directly as needed.
  3. Added support for a new type of registry called the "leader registry" which leverages a user-provided leader election mechanism to elect a node as the registry leader and then that node runs the in-memory version of the registry. This allows users to take advantage of advanced registry features like intelligent load balancing with no dependency on complex systems like FoundationDB. Also added support for making the leader registry handle leader transitions intelligently by letting nodes fills the new registry leaders state automatically (when they perform ensureActivation() calls to refresh their cache) from their cache of the previous leaders state.
  4. Added the ability to load balances actors automatically not just based on count, but also based on memory usage. By default the registry will now try to rebalance actors when the delta in terms of actor memory usage between nodes is > 2GiB.
  5. Added a new activations_cache.go implementation that allows clusters running with registry implementations that do not care about linearizability to survive arbitrarily long failures of the registry for any actors whose current activation is cached in the calling node.
  6. Added a semaphore for registry ensureActivation() calls to prevent nodes from DDOSing the registry during pathological workloads.

Most interesting files to review:

  1. leader_registry.go
  2. examples/leaderregistry/main_test.go
  3. actor_resource_tracker.go
  4. changes to kv_registry.go