Improve scheduling algorithm

[shonfeder]

We often see the current scheduling to result in one worker getting overloaded with a massive queue due to cache hints while other workers are starved for work.

We want the job scheduling algorithm to take cache hints into account, but also to consider worker capacity and availability. There are times where this should be able to really improve our thru put.

Related to, but more general than #168

[talex5]

We want the job scheduling algorithm to take cache hints into account, but also to consider worker capacity and availability.

It should already be doing that:

https://github.com/ocurrent/ocluster/blob/a27ef61876471fa5620d2422fef441bf50d0260f/scheduler/pool.ml#L368-L373

However, there are a few problems (or were, last time I looked):

• The estimates of the times for cold vs warm caches are hard-coded and arbitrary: https://github.com/ocurrent/ocluster/blob/a27ef61876471fa5620d2422fef441bf50d0260f/scheduler/cluster_scheduler.ml#L33-L36
• The cache may have been cleared on the worker, but the scheduler doesn't know.
• If a machine gets stuck but remains connected then the scheduler will wait indefinitely for it do to the work it claims it can do, rather than deciding that it's broken and reassigning (though if the worker is well enough to be performing health-checks then it can pause itself, which fixes the problem).