belforte
commented
2 years ago In the end the explanation of all the issues I had with environment is a (big) initial misunderstanding.
I thought that the job payload is executed inside Scram() in a clean environment, just since it was done that way (but in the COMP) environment in the past.
Instead as @khurtado explained
cmsRun is executed through a subprocess call (+ a wrapper for the CMSSW setup) and not with Scram() , those variables are not lost (which is why the WM PYTHONPATH is removed by hand before executing it, but then put back for other stuff that needs it afterwards in the executor)
Also, quoting from https://github.com/dmwm/WMCore/wiki/Notes-about-environment-variables-passed-to-the-Scram-environment-or-modified-when-running-the-CMSSW-executable , :
The CMSSW executable (cmsRun) is not executed directly by the executor with Scram(). Instead, it runs a wrapper script that setups its own environment (with a name stepName-main.sh. E.g.: cmsRun1-main.sh). This bash script is written on the fly and defined HERE.
Since this wrapper script is setting up its own scram environment, the script itself is called by simply using a subprocess call. However, some environment variables need to be overridden in order to avoid problems between the CMSSW environment and the WM environment.
The environment override is defined HERE and basically make sure that:
- The WM PYTHONPATH is not passed in the subprocess call.
_- It sets XRD_LOADBALANCERTTL to workaround a problem at CERN related to the GSI authentication plugin and EOS with XRootD
- It sets the HOME environment variable_
After all these changes in the environment, the CMSSW executable is invoked through this wrapper script HERE However, since we clean up the WM PYTHONPATH in the os system, other steps (e.g.: in the stepChain workflow) would fail after this if they can't find the WM libraries, so the original WM PYTHONPATH is put back in the environment after calling the cmssw executable/cmsRun wrapper.
So this is significantly different from current CRAB approach, and we need to decide if to change CRAB Job Wrapper to follow more closely what WMCore does and how much closely since we do not want to reproduce all WMA Step machinery.
Maybe as simple as use Scram(envCmd=...) ( from HERE ) to cleanup $PYTHONPATH and then run Scram() in the job start environment ?
@amaltaro @khurtado @mapellidario @dciangot your input is more than welcome !
By the way, currently CRAB does nothing about
set XRD_LOADBALANCERTTL to workaround a problem at CERN related to the GSI authentication plugin and EOS with XRootD
how much worried should we be ? Does anybody know what this thing is ? Is there any document/ticket/issue/elog about it ?
khurtado
mapellidario
see https://github.com/dmwm/WMCore/issues/10970#issuecomment-1039742503 and especially https://github.com/dmwm/WMCore/wiki/Notes-about-environment-variables-passed-to-the-Scram-environment-or-modified-when-running-the-CMSSW-executable