search

BSC-Support-Team / GREASY

Greasy is a tool designed to make easier the deployment of embarrassingly parallel simulations in any environment.
Other
11 stars 5 forks source link

Add the @ workdir syntax #3

Closed victorusu closed 3 years ago

victorusu commented 3 years ago

This introduces the [@ workdir @] syntax. In this syntax the users can change dir just before the execution of a given task. The feature may seem strange at first because users can easily add tasks that have cd commands before the executable call. However, the cd command does not work for basic engine combined with the srun method. Therefore it is necessary to introduce such feature.

All engines and run methods implement this feature, therefore it is easy for the end user to migrate task files between greasy installation that use different engines.

Additionally, the logic of emitting srun commands for the basic engine is slightly different from the original code. In this implementation one always emit the srun command even when executing tasks in the master node. This is due to the fact that SLURM allows for SPANK plugins to be defined for each job step (srun call). In order to avoid that some tasks are executed as job steps and others not, then one needs to always invoke thes tasks using the srun command.

In practice, the task file can look like

[@ $HOME/01 @]  hostname && pwd
[@ $HOME/02 @]  hostname && pwd
...

where $HOME/01 and $HOME/02 are distinct folders where the command hostname && pwd will be executed.

victorusu commented 3 years ago

@PabloRodenas, could you please take a look at this PR? What do you think?

PabloRodenas commented 3 years ago

Hi Victor, sorry for the late reply. It looks quite interesting improvement although before confirming the PR I need to look (and test) deeper into this change. I did a fast view on Friday but this is being a busy week. We really appreciate your contribution. Thanks!

PabloRodenas commented 3 years ago

Here we are. So finally I got the time for reviewing in deep this request.

I've checked and tested and everything seems to be OK, the change do not disturb on using dependencies so it is possible to combine both tags for a task.

Only mention that I have to add -std=c++11 as cxxflags in src/Makefile.am in order to tell the compiler (tested on intel/2017) what to do with the word 'auto'. I guess this is our particular case, so I will commit this change after accepting the PR.

Again, thanks for your interest in GREASY and apologises for the delay on accepting this PR.

victorusu commented 3 years ago

@PabloRodenas, thanks!