search

CrayLabs / SmartSim

SmartSim Infrastructure Library.
BSD 2-Clause "Simplified" License
219 stars 36 forks source link

Load environment modules before launching an `Experiment` #369

Open b-fg opened 9 months ago

b-fg commented 9 months ago

Description

Specially in clusters, sometimes the environmental modules needed for the execution of an Experiment are incompatible with the modules needed for SmartSim and SmartRedis. It would be useful to be able to load modules in the experiment subshell before the run command is launched.

Justification

HPC cluster users, specially, would benefit from this feature.

Implementation Strategy

To be able to pass a bash script which is called before the Experiment.start method in the same subshell. The bash script may contain the environmental modules needed for the executable.

b-fg commented 9 months ago

Because of compatibility issues, I am currently using SmartSim v0.4.2 in the cluster. It would be appreciated if someone can point me to a quick fix or a way to do this in that version too. Thanks!

b-fg commented 9 months ago

Regarding SmartSim v0.4.2, I managed to get this working by passing batch_settings when creating a model and use the add_preamble method to source the relevant bash file that contains the modules. The downside of this of course is that everytime a model is restarted a new job needs to be allocated, and this can slowdown the execution. Ideally, I would like to find a method to load the modules in the sub-shell of the model run command, so I only need to submit a single job with the necessary resources.