LourensVeen
commented
5 years ago All components need to run inside of a single allocation, so the basic idea is to submit a job to the scheduler with enough resources to run the whole simulation. Within that job, the basic functionality is to first start the manager, passing it the yMMSL file, and to get its location from its standard output. Then start the compute element instances, passing them the location of the manager.
This means that something is needed to start the instances on specific nodes/cores within the allocation, e.g. srun, an srun wrapper, or a pilot job framework. And we need to have information on how many resources each instance needs. My idea here is to extend yMMSL with that information, using a third top-level attribute (besides the model and the settings). It would probably be a list of compute elements (or perhaps instances, or the ability to specify either) with for each the required resources. Number of cores is the primary requirement I'd say, things like RAM and disk space could be added later.
The launcher would then read that information, add it up to calculate the total number of cores, and allocate that from the scheduler. Within that allocation another component would read it again and start each component as described above.
So the algorithm for the launcher would look something like this, I think:
- Read yMMSL file
- Determine total number of cores required
- Determine which implementations to run
- Stage yMMSL file
- Stage implementations (if not already installed)
- Submit job
Then within the allocation:
- Read yMMSL file
- Determine which implementations to run with how many resources
- Start launcher, passing yMMSL file
- Start implementations, passing manager location
Submodels may require additional inputs in the form of data files, and they may produce output files. There has to be a description of those as well somewhere, so that they can be staged in and out, but I'm not sure that the yMMSL file is the right place to do so. How does FabSim3 deal with that currently? And how do you specify the amount of required resources?
djgroen
I'm planning to make a launcher for MUSCLE3 in the form of a FabSim3 plugin, to make it easier to start and coordinate submodels, especially when they are launched remotely, or in distributed fashion.
Before I make a first version of such a launcher, I would to ask for any specific design preferences. @LourensVeen are there particular preferences from your side in regards to how I design or construct such a launcher, or what kind of functionalities it should support?
I plan to keep this issue for design discussions and feedback, while moving any technical discussions and other tasks to the respective plugin repository. But if you would like to organize it otherwise, let me know and I'll rearrange things.