Open ms705 opened 9 years ago
My feeling is that option 2 is the right way to do it. However, it depends on how much re-factoring we would have to do for the change. If it is relatively difficult, then we can go for option 1. It looks like a good in-between solution: not too-hacky and not difficult to get in.
Option 3 is the least appealing to me because it doesn't simplify at all the code of the cost models. We would still have to do all the tests to see if a task is active. Moreover, we would have additional knobs to twist (e.g., interval at which to GC tasks, number of inactive tasks stored before GC is triggered).
We currently keep track of all tasks known to a coordinator in the
TaskMap_t
data structure owned by theCoordinator
. This contains tasks in new, runnable, running, completed, failed and various other states. We use it for the web UI, scheduling and the management of task-specific data structures.However, the flow graph (and, consequently, the cost models) sometimes needs to iterate over all tasks that are currently of interest to the scheduler (i.e., those which are still eligible for scheduling: runnable, running and failed ones), and can get tripped up by "archived" tasks that are still in the task map.
In order to increase the efficiency of such iterations and clear up the semantics, we should de-conflate the two purposes of the task map. There are several options for this:
Interested in views on what the best way forward is.