Feature request: elastic execution pool

[lgray]

Related to #900

It should be possible to expand and contract the number of workers attached to a parsl session. Right now it seems that we can only expand using the 'simple' strategy, and in a very strange way (where it almost immediately goes to max_blocks).

For instance:

• set a target time for the whole processing workflow to complete
• accumulate statistics for N jobs (N~100), use this to estimate time to completion
• if time to completion is > M sigma (M ~2-3) from target time, submit enough managers to bring the expected completion time to the target time.
• If execution is finished (empty task queue) for a manager for > X minutes (X ~10) remove that manager from the pool.
  • do not remove manager if number of managers < min managers

[benclifford]

Maybe it would be interesting to expose the strategy interface to parsl users rather than using the hard-coded set of strategies now. Some raw data is collected through monitoring which might support other interesting strategies.