Is your feature request related to a problem? Please describe.
In parsl at 06325a518510d7ffec2ed1703fdd974a94c67653 there are at least three different ways in which an executor can indicate that it wants core parsl to call scale up/scale down operations on it; the managed property, the scaling_enabled property and inheritence from BlockProviderExecutor.
These are all fairly loosely described and implemented.
Describe the solution you'd like
I would like to consolidate these so that inheriting from BlockProviderExecutor is the indicator that scaling should be done - initially by the existing scaling code launched from the DFK, but later, by moving the implementation of block scaling into the BlockProviderExecutor.
Additional context
These several ways of indicating a fairly vague notion of what should and should not be done has made the code awkward to type-check in the benc-mypy branch - why should, for example, the ThreadPoolExecutor have attributed related to providers, which are only used in block scaling executors?
Potentially other executors (for example, as demonstrated in the funcx hard-fork of htex) want some kind of scaling, but the core parsl scaling code is probably insufficient (for example, the funcx htex recognises different kinds of workers, based on containers).
Is your feature request related to a problem? Please describe.
In parsl at 06325a518510d7ffec2ed1703fdd974a94c67653 there are at least three different ways in which an executor can indicate that it wants core parsl to call scale up/scale down operations on it; the
managed
property, thescaling_enabled
property and inheritence fromBlockProviderExecutor
.These are all fairly loosely described and implemented.
Describe the solution you'd like
I would like to consolidate these so that inheriting from
BlockProviderExecutor
is the indicator that scaling should be done - initially by the existing scaling code launched from the DFK, but later, by moving the implementation of block scaling into the BlockProviderExecutor.Additional context
These several ways of indicating a fairly vague notion of what should and should not be done has made the code awkward to type-check in the
benc-mypy
branch - why should, for example, the ThreadPoolExecutor have attributed related to providers, which are only used in block scaling executors?Potentially other executors (for example, as demonstrated in the funcx hard-fork of htex) want some kind of scaling, but the core parsl scaling code is probably insufficient (for example, the funcx htex recognises different kinds of workers, based on containers).