dchackett
commented
6 years ago The current scheme unifies Runners and Tasks. In order to have access to that runner, the run-specification script just imports the appropriate module. When taxis are run, they reimport everything that the run-specification script had imported. So: no annoying file management required to use different runners, and users don't need to modify anything in the taxi source to use their own runners.
Ideally, taxi would be able to pop in and out modules for various code suites (e.g., (multirep) MILC vs. ...)
One structural issue, at present, is the runner scripts. Different applications will require different runner scripts. A minimal modularization is just to move runners/ -> runners/mrep_milc/, and point at this subfolder in run-specification scripts.
However: tasks.py is mostly, at present, Task classes that are specific to multirep MILC. These tasks are specifically aware of the runner scripts that they correspond to. So, any modularization of the runners should also involve a modularization of tasks. How can we modularize these together in a graceful way?
Idea 1: Replace runners/ with applications/ (or apps/ or something snappier). Inside (e.g.) apps/mrep_milc/ is an application-specific mrep_tasks.py. There is a subfolder apps/mrep_milc/runners/.
Issue: Without allowing "application mixing", we will inevitably have some code duplication. For instance, for the multirep project, we use the multirep MILC HMC binary, but "MilcI9" clover_invert and flow binaries. We might, in theory, have a separate application folder like "apps/MilcI9" to run HMC with the single-rep MilcI9 binary. The clover_invert/flow runner scripts would have to be duplicated here, if we just want to simply pop app folders in and out.
Possible solution: Have separate runners/classes for multirep MILC and MilcI9 HMC binaries in a unified "apps/milc", similar to how we need separate runners/classes for HMC and pure-gauge.
Idea 2: Runner scripts include their corresponding Task classes.
Issue 2-1: Runner scripts must be Python 2.6 compatible. At present, this restriction isn't necessarily true for task classes.
Issue 2-2: Aesthetically, the runner scripts should be minimal and even useful outside the context of taxi.
Issue 2-3: Including all the right tasks becomes tedious. An aesthetically pleasant solution to this issue could be: specific applications (e.g., milcI9, mrep_milc) are implemented by some importable file that imports all the corresponding tasks from the runner scripts.