Change the load balancer to reduce invoker activation queueing

[Ahmed-elsayed-mahmoud]

Problem statement:

The current workflow of queueing activations to per-invoker topics has some limitations like: early routing: invokers are all treated the same, and once an invoker is selected to process an activation, it is queued for that invoker, even though a different invoker may end up having more capacity to process it earlier.

Suggested solution:

Provide a unified overflow topic and route an overflowed activation (all invokers reach execution capacity) to an invoker ONLY after an invoker has become available to process the activation. This provides benefits like: queuing behavior is predictable ("waiting for execution", instead of "waiting for capacity" + "waiting for execution") since invokers won't be subjected to loads they cannot handle, and scheduling activations to a specific invoker will be more fair.

[markusthoemmes]

Thanks for the proposal 🎉 . What you're suggesting has indeed been discussed on the dev list and I think is agreed upon. See https://cwiki.apache.org/confluence/display/OPENWHISK/Invoker+Activation+Queueing+Change for the writeup of the initial proposal.

[Ahmed-elsayed-mahmoud]

@markusthoemmes Thanks for your comment! I have already finished implementation of this proposal and I will submit a pull request soon.