Parsl / parsl

Parsl - a Python parallel scripting library
http://parsl-project.org
Apache License 2.0
503 stars 195 forks source link

Consolidate executor API to handle block scaling #2539

Open benclifford opened 1 year ago

benclifford commented 1 year ago

Is your feature request related to a problem? Please describe.

In parsl at 06325a518510d7ffec2ed1703fdd974a94c67653 there are at least three different ways in which an executor can indicate that it wants core parsl to call scale up/scale down operations on it; the managed property, the scaling_enabled property and inheritence from BlockProviderExecutor.

These are all fairly loosely described and implemented.

Describe the solution you'd like

I would like to consolidate these so that inheriting from BlockProviderExecutor is the indicator that scaling should be done - initially by the existing scaling code launched from the DFK, but later, by moving the implementation of block scaling into the BlockProviderExecutor.

Additional context

These several ways of indicating a fairly vague notion of what should and should not be done has made the code awkward to type-check in the benc-mypy branch - why should, for example, the ThreadPoolExecutor have attributed related to providers, which are only used in block scaling executors?

Potentially other executors (for example, as demonstrated in the funcx hard-fork of htex) want some kind of scaling, but the core parsl scaling code is probably insufficient (for example, the funcx htex recognises different kinds of workers, based on containers).

benclifford commented 1 year ago

tag @yadudoc who is doing some work in this space too with funcX