radical-cybertools / radical.pilot

RADICAL-Pilot
http://radical-cybertools.github.io/radical-pilot/index.html
Other
54 stars 23 forks source link

Enforce task env within `exec.sh` script? #3039

Closed mtitov closed 11 months ago

mtitov commented 1 year ago

Should we prepare task environment as we do for launcher environment, but use preserved RP's virtual environment

https://github.com/radical-cybertools/radical.pilot/blob/cf7c159246a45bbec81163facf6bec984e68bb96/src/radical/pilot/agent/executing/popen.py#L642-L644

Thus, if user doesn't provide named_env, then we activate RP's env

    ret += '\n# named environment\n'
    if td['named_env']: 
        ...
    else:
        ret += '. $RP_PILOT_SANDBOX/env/rp_named_env.rp.sh\n'
andre-merzky commented 11 months ago

The first part would make sense (prepare the env). I would suggest not to fall back to the RP env. I like that we make the promise that the task runs by default in the virgin environment as found on the nodes when the bootstrapper lands. That is the least likely env to conflict with user module load and export directives IMHO. Also, we would have no means anymore to use that clean env for tasks. The user can always specify rp if needed (but we need to document that it could interfere with the RP agent itself).