cms: need to set cms environment manually for serial workflow

[dprelipcean]

Using the cms docker image to do the cms-reco example, I am following their instructions on how to prepare the environment to run the analysis. One of them includes the command scram b, which fails as it cannot be found.

$ cat reana.yaml
version: 0.5.0
inputs:
  parameters:
    repo: DimuonSpectrum2011
workflow:
  type: serial
  specification:
    steps:
      - environment: 'cmsopendata/cmssw_5_3_32'
        commands:
        - cd WorkDir/repo && scram b
outputs:
  files:
    - DoubleMu.root
$ reana-client run
[INFO] Creating a workflow...
workflow.1
[INFO] Uploading files...
[INFO] Starting workflow...
workflow.1 is running
$ reana-client logs -w workflow.1

JOB LOGS
job id: 0b139916-e7a6-42ef-95b3-5d5647ca8d02
bash: scram: command not found

This can be reproduced locally as follows. If the cmsuser is not modified, everything works as expected.

$ docker run --name opendata -it --rm cmsopendata/cmssw_5_3_32 /bin/bash
Setting up CMSSW_5_3_32
CMSSW should now be available.
[17:09:45] cmsusr@0306021bf0d7 ~/CMSSW_5_3_32/src $ id
uid=1000(cmsusr) gid=1000(cmsusr) groups=1000(cmsusr),0(root)
[17:09:45] cmsusr@0306021bf0d7 ~/CMSSW_5_3_32/src $ scram
***************************************************
SCRAM HELP ------------- Recognised Commands
***************************************************

     scram arch
     scram build
     scram config
     scram db
     scram list
     scram project
     scram runtime
     scram setup
     scram tool
     scram unsetenv
     scram version

See scram manual pages for detail documentation about these commands.

If the user is changed to belong to the root group (0), as reproduced here, it does not work.

$ docker run --name opendata -it --rm -u root cmsopendata/cmssw_5_3_32 /bin/bash
Setting up CMSSW_5_3_32
CMSSW should now be available.
[17:12:07] root@0e2aa926d33a ~/CMSSW_5_3_32/src $ usermod cmsusr -g 0
[17:12:18] root@0e2aa926d33a ~/CMSSW_5_3_32/src $ su cmsusr
[17:12:26] cmsusr@0e2aa926d33a ~/CMSSW_5_3_32/src $ scrum
bash: scrum: command not found
[17:12:29] cmsusr@0e2aa926d33a ~/CMSSW_5_3_32/src $ id
uid=1000(cmsusr) gid=0(root) groups=0(root)
[17:12:31] cmsusr@0e2aa926d33a ~/CMSSW_5_3_32/src $

One attempt to fix it was to use the absolute path. Running locally which scram, I could then input in the yaml file instead of scram, the absolute path one: /opt/cms/common/scram. However, other errors occured:

job id: 220ada14-280f-4d90-a797-d6346550b375
/opt/cms/common/scram: line 2: cmsarch: command not found
cat: /opt/cms//etc/default-scramv1-version: No such file or directory
Unable to find SCRAM version  for  architecture.

Investigating even more via cat scram, the command that fails is cat ${srbase}/etc/default-scram/${scram_rel_series}, which when run locally is evaluated to /etc/default-scram/, but when run through reana it gives /opt/cms//etc/default-scramv1-version

[katilp]

@dprelipcean Thanks for reporting! There is then cmsenv missing, i.e. the command sequence should be:

cd WorkDir/repo
cmsenv
scram b

But actually, for the re-reconstruction scram b is not needed, as it would compile the code but in this case there are no changes needed for the precompiled executable which can be run with cmsRun. But cmsenv is needed, so it should be

cd WorkDir/repo
cmsenv

[dprelipcean]

@katilp Thank you for the hint!

I tried to work around it via absolute path, i.e. run locally which cmsenv and then input that into the reana.yaml file, but I got which: no cmsenv. More investigation needed.

[katilp]

OK, pinging @clelange for more information. The container has the CMS environment (i.e. the release area CMSSW_5_3_32) so maybe then cmsenv is not needed. Does it run without (and without compiling with scram b which is not needed in this example?

[clelange]

@dprelipcean Could it be you just have a typo above in the command? You run scrum instead of scram after adding cmsusr to group 0:

[17:12:26] cmsusr@0e2aa926d33a ~/CMSSW_5_3_32/src $ scrum
bash: scrum: command not found

[katilp]

@clelange For my own understanding, is cmsenv needed at all in the container, or is the environment set already?

[dprelipcean]

@dprelipcean Could it be you just have a typo above in the command? You run scrum instead of scram after adding cmsusr to group 0:

[17:12:26] cmsusr@0e2aa926d33a ~/CMSSW_5_3_32/src $ scrum
bash: scrum: command not found

That is correct, I've tried again and it was just a typo. My apologies!

I think that the error is related to the fact that reana uses the normal shell, and not the cms shell.

[clelange]

@clelange For my own understanding, is cmsenv needed at all in the container, or is the environment set already?

We have an entrypoint script that sets up the CMS environment. You overwrite/bypass it in your workflow, so you just need to run the commands listed in there to get the environment.

[katilp]

@dprelipcean NB that the re-reconstruction task needs access run-time to the condition database and as it is now, this is achieved with

ln -sf /cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1_RUNA FT_53_LV5_AN1
ln -sf /cvmfs/cms-opendata-conddb.cern.ch/FT_53_LV5_AN1_RUNA.db FT_53_LV5_AN1_RUNA.db

before the cmsRun command.

Did I understand correctly from @tiborsimko that this area is now accessible through ReANA ? If the container cmsopendata/cmssw_5_3_32 is used as in http://opendata.cern.ch/docs/cms-guide-docker this db is not accessible, but my knowledge is too limited to understand if it can be accessed using this same container in the ReANA context...

[dprelipcean]

@clelange thank you for the input! I'm now using entrypoint script inside the analysis commands to set up the environment, per your instructions, e.g. bash entrypoint.sh cmsenv, however unsuccessfully.

The logs that I get in return are:

Setting up CMSSW_5_3_32
WARNING: In non-interactive mode release checks e.g. deprecated releases, production architectures are disabled.
WARNING: There already exists /var/reana/users/00000000-0000-0000-0000-000000000000/workflows/89bbc792-b905-4cb4-9a46-33708599206d/CMSSW_5_3_32 area for SCRAM_ARCH slc6_amd64_gcc472.
CMSSW should now be available.
entrypoint.sh: line 11: exec: cmsenv: not found

Similarly for scram and cmsRun.

[tiborsimko]

@katilp Yes, the condition database on CVMFS can be accessed with any container, the only requirement is that the user should specify the necessary CVMFS volumes to be live-mounted in the reana.yaml resource section: https://reana.readthedocs.io/en/latest/userguide.html#declare-necessary-resources

[clelange]

@dprelipcean There is already a CMSSW release, so you don't need to call scramv1 project CMSSW (which is aliased to cmsrel). All you need to do is change to CMSSW_5_3_32/src and execute cmsenv. This again, is an alias, to make it work you might have to do shopt -s expand_aliases, or, to give you all details, see below.

which cmsenv
cmsenv='eval `scramv1 runtime -sh`'
which cmsrel
cmsrel='scramv1 project CMSSW'
        /cvmfs/cms.cern.ch/common/scramv1

This means:

cd CMSSW_5_3_32/src
eval `scramv1 runtime -sh`

cmsRun is an executable (not aliased)

[dprelipcean]

I encounter this issue only when using the serial workflow, as for cwl one can specify the base command as /bin/zsh and then everything is set accordingly. So I will focus on the cwl workflow for the moment and possibly return to the serial one afterwards.

Nevertheless, thanks to your help @clelange, I have managed to get it working, e..g: cd ./CMSSW_5_3_32/src/WorkDir/${repo} && source /opt/cms/cmsset_default.sh && scram b

P.S.: Have transferred the issue to its proper place as it's not what we initially though it is.

[dprelipcean]

Solved temporarily with:

      source /opt/cms/cmsset_default.sh ;\
      scramv1 project CMSSW CMSSW_5_3_32 ;\
      cd CMSSW_5_3_32/src ;\
      eval `scramv1 runtime -sh` ;\

before the cms specific commands and openend an issue to deal with how kubernetes overwrites the entrypoint when running the docker images.

Closing this one.