Stability and scalability fixes; expose tuning parameters

[chungers]

This PR

• Fixed deadlocks in simulator code that prevented simulated caps from working
• Added simulator control parameters for adding provision delays to simulate long-running or slow instance plugin Provision calls.
• Exposes tuning parameters and updated example / docs (workers.yml) to show tuning parameters when provisioning large set of resource (> 500 simulated instances).
• Make provision and destroy asynchronous with deadlines.
• Using example docs/controller/pool/workers.yml and simulator, it is now possible to provision > 500 instances and very quick scale up and scale down (before settling into steady state between commits). More examples, docs to come.

Signed-off-by: David Chung david.chung@docker.com

[codecov[bot]]

Codecov Report

Merging #898 into master will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #898   +/-   ##
=======================================
  Coverage   48.56%   48.56%           
=======================================
  Files          84       84           
  Lines        7837     7837           
=======================================
  Hits         3806     3806           
  Misses       3721     3721           
  Partials      310      310

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8e2ce7f...94b71f7. Read the comment docs.