search

flux-framework / flux-sched

Fluxion Graph-based Scheduler
GNU Lesser General Public License v3.0
84 stars 39 forks source link

implement partial release of resources #1151

Open garlick opened 2 months ago

garlick commented 2 months ago

Problem: as discussed in flux-framework/flux-core#4312, the original plan for partial release of resources was to give the scheduler a free RPC for each R fragment of a job's resources that can be returned to the pool. In fluxion, the R is ignored in the free callback and the jobid is used instead to free all resources allocated to the job.

An additional problem is that flux-core cannot fragment the contents of the opaque scheduling key in R.

Assuming we figure out a way in flux-core to release resource in parts, how can this be made to work in fluxion?

Note that RFC 27 would need to be updated as it currently describes a single free RPC.

garlick commented 2 months ago

A fragment would contain all the resources allocated to the job on one or more execution targets (broker ranks). That is, it would not be further subdivided.

One thought on the JGF problem is that perhaps a combination of job ID and the list of execution target ids from Rv1 would be sufficient to identify the resources being freed.

milroy commented 1 month ago

With the assumption that a fragment contains a subset of the job's broker ranks but the entire R (i.e., the full R for each broker rank) for each fragment broker rank, adding this support should be straightforward.

Mainly what's needed is to identify the broker ranks in the R fragment and iterate through the vertices in the by_rank graph metadata map for each rank. Then remove the scheduling and planner data per vertex. Updating the vertices' ancestors' pruning filters will require some thought, though...