search

riga / cms-hlt-parser

Read, parse and provide CERN CMS High-Level Trigger and Luminosity information.
1 stars 1 forks source link

Basic installation and examples not working #6

Closed trtomei closed 1 year ago

trtomei commented 1 year ago

Hi Marcel, Clemens,

First, thanks for this development! I think this is a very useful tool, and I plan to use it to compile some information regarding triggers during Run 1 and 2.

Unfortunately, I cannot get it to work. :( On a lxplus8 node, I followed the initial instructions and tried the first example... it gives me:

tomei@lxplus815:cms-hlt-parser $ law run hltp.GetLumiData     --hlt-path 'HLT_IsoMu2?_v*'     --print-summary

INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 836915] Worker Worker(salt=4747989597, workers=1, host=lxplus815.cern.ch, username=tomei, pid=836915) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)

=============================== entering sandbox ===============================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

Traceback (most recent call last):
  File "/afs/cern.ch/cms/lumi/brilconda-1.1.7-cc7/bin/pip", line 4, in <module>
    import pip
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.6/site-packages/pip/__init__.py", line 6
    def main(args: Optional[List[str]] = None) -> int:
                 ^
SyntaxError: invalid syntax

=============================== leaving sandbox ================================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

Some information that might be useful:

$ uname -a
Linux lxplus815.cern.ch 4.18.0-477.13.1.el8_8.x86_64 #1 SMP Thu May 18 10:27:05 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
$ which python3
/usr/bin/python3
$ python3 --version
Python 3.6.8
$ which pip
~/work/HEP/CMS/TSG/cms-hlt-parser/software/bin/pip
$ pip --version 
pip 21.3.1 from /afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.6/site-packages/pip (python 3.6)
echo $PYTHONPATH
/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser:/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.6/site-packages:

Cheers, Thiago

trtomei commented 1 year ago

Hi,

So, after some private chat with Clemens, I tried to use a more recent python version, but still doesn't work...

 $ law run hltp.GetLumiData     --hlt-path 'HLT_IsoMu2?_v*'     --print-summary

INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 125014] Worker Worker(salt=2409173171, workers=1, host=lxplus817.cern.ch, username=tomei, pid=125014) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)

=============================== entering sandbox ===============================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

ImportError: No module named site
Collecting brilws
  Downloading brilws-3.7.4.tar.gz (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.5/52.5 kB 6.2 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: brilws
  Building wheel for brilws (setup.py) ... done
  Created wheel for brilws: filename=brilws-3.7.4-py3-none-any.whl size=63996 sha256=6dd2b7d1a42c49244c3e918d846751f7cb8e8cd3f9692b65164f68e91e09fa82
  Stored in directory: /tmp/tomei/pip-ephem-wheel-cache-2qc6ykoe/wheels/ce/9e/29/48d51b148196f16188afa4d18ebdb02d75518571de67fa77fe
Successfully built brilws
Installing collected packages: brilws
Successfully installed brilws-3.7.4
ImportError: No module named site
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 125073] Worker Worker(salt=2409173171, workers=1, host=lxplus817.cern.ch, username=tomei, pid=125014) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
running brilcalc lumi ...
cmd: brilcalc lumi -u /pb -i /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt --normtag /cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json --hltpath "HLT_IsoMu2?_v*"
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/bin/brilcalc", line 5, in <module>
    from brilws.cli.brilcalc_main import brilcalc_main
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/brilws/cli/__init__.py", line 1, in <module>
    from brilws.cli.brilcalc_main import brilcalc_main
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/brilws/cli/brilcalc_main.py", line 8, in <module>
    import schema
ModuleNotFoundError: No module named 'schema'
failed (took 0.11 seconds)
ERROR: [pid 125073] Worker Worker(salt=2409173171, workers=1, host=lxplus817.cern.ch, username=tomei, pid=125014) failed    hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 386, in call
    return fn(task, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 519, in localize
    return fn(task, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/hltp/tasks.py", line 102, in run
    out = self.call_step(cmd, "running brilcalc lumi ...", publish_cmd=True)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/hltp/base.py", line 88, in call_step
    raise Exception("{} failed".format(prog or cmd.split(" ", 1)[0]))
Exception: brilcalc failed
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   FAILED
INFO: This progress looks :( because there were failed tasks

=============================== leaving sandbox ================================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

I now have:

$ which python3
/cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/bin/python3
$ python3 --version
Python 3.9.12
$ pip --version
pip 23.2.1 from /afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/pip (python 3.9)
$ echo $PYTHONPATH
/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser:/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages:/cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/lib/python3.9/site-packages/itk:/cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/lib:/cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/lib/python3.9/site-packages

Cheers, Thiago

trtomei commented 1 year ago

Finally, also with python 3.11 from conda:

$ law run hltp.GetLumiData     --hlt-path 'HLT_IsoMu2?_v*'     --print-summary

INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 3085296] Worker Worker(salt=190736611, workers=1, host=lxplus812.cern.ch, username=tomei, pid=3085296) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)

=============================== entering sandbox ===============================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

Traceback (most recent call last):
  File "/afs/cern.ch/cms/lumi/brilconda-1.1.7-cc7/bin/pip", line 4, in <module>
    import pip
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.11/site-packages/pip/__init__.py", line 6
    def main(args: Optional[List[str]] = None) -> int:
                 ^
SyntaxError: invalid syntax

=============================== leaving sandbox ================================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================
clelange commented 1 year ago

Same for me, tried LCG 104 view, 104 Python release as well as CMSSW_13_0_10. The packaging with law causes too much trouble here. It's probably easier to just run the commands individually.

riga commented 1 year ago

@trtomei Do you maybe have something in your bashrc/zshrc that changes the default lxplus8 environment? I just tried your command on a fresh lxplus8 machine and it works.

The packaging with law causes too much trouble here.

@clelange What do you mean?

trtomei commented 1 year ago

@riga I checked my bashrc and I have only a bunch of aliases.

Using law3 also failed. See the sandbox error here:

=============================== entering sandbox ===============================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

ImportError: No module named site
Collecting brilws
  Downloading brilws-3.7.4.tar.gz (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.5/52.5 kB 6.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: brilws
  Building wheel for brilws (setup.py) ... done
  Created wheel for brilws: filename=brilws-3.7.4-py3-none-any.whl size=63996 sha256=9b6f5ee1fa916499240b923fbb6daf837e10597c590419572658d5c2832574fa
  Stored in directory: /tmp/tomei/pip-ephem-wheel-cache-nek2v2rx/wheels/ce/9e/29/48d51b148196f16188afa4d18ebdb02d75518571de67fa77fe
Successfully built brilws
Installing collected packages: brilws
Successfully installed brilws-3.7.4
ImportError: No module named site
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 972882] Worker Worker(salt=5688569520, workers=1, host=lxplus812.cern.ch, username=tomei, pid=972791) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
running brilcalc lumi ...
cmd: brilcalc lumi -u /pb -i /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt --normtag /cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json --hltpath "HLT_IsoMu2?_v*"
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/bin/brilcalc", line 5, in <module>
    from brilws.cli.brilcalc_main import brilcalc_main
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/brilws/cli/__init__.py", line 1, in <module>
    from brilws.cli.brilcalc_main import brilcalc_main
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/brilws/cli/brilcalc_main.py", line 8, in <module>
    import schema
ModuleNotFoundError: No module named 'schema'
failed (took 0.15 seconds)
ERROR: [pid 972882] Worker Worker(salt=5688569520, workers=1, host=lxplus812.cern.ch, username=tomei, pid=972791) failed    hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 386, in call
    return fn(task, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 519, in localize
    return fn(task, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/hltp/tasks.py", line 102, in run
    out = self.call_step(cmd, "running brilcalc lumi ...", publish_cmd=True)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/hltp/base.py", line 88, in call_step
    raise Exception("{} failed".format(prog or cmd.split(" ", 1)[0]))
Exception: brilcalc failed
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   FAILED
INFO: This progress looks :( because there were failed tasks

=============================== leaving sandbox ================================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

This is using LCG_102b.

riga commented 1 year ago

@trtomei I just managed to reproduce the issue (looks like my environment had some left-overs from a bril test on python 3 a while back) and get back to you later today with a fix.

riga commented 1 year ago

@trtomei The issue came from the fact that bril 1.1.7 from /afs was still using python 2. I updated the code to use /cvmfs/cms-bril.cern.ch/brilconda3 instead and now it should work. Can you try again with a fresh checkout? Thank you!

trtomei commented 1 year ago

@riga No dice... again, using the LCG102b, but it still has problems with the python from brilconda

=============================== entering sandbox ===============================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
Collecting brilws
  Using cached brilws-3.7.4-py3-none-any.whl
Installing collected packages: brilws
Successfully installed brilws
Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 1525350] Worker Worker(salt=6439333770, workers=1, host=lxplus800.cern.ch, username=tomei, pid=1524837) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
running brilcalc lumi ...
cmd: brilcalc lumi -u /pb -i /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt --normtag /cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json --hltpath "HLT_IsoMu2?_v*"
Error in sitecustomize; set PYTHONVERBOSE for traceback:
FileNotFoundError: [Errno 2] No such file or directory: '/cvmfs/LCG_externals_brilconda-3.2.16.txt'
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/bril/bin/brilcalc", line 5, in <module>
    from brilws.cli.brilcalc_main import brilcalc_main
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/bril/lib/python3.7/site-packages/brilws/cli/__init__.py", line 1, in <module>
    from brilws.cli.brilcalc_main import brilcalc_main
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/bril/lib/python3.7/site-packages/brilws/cli/brilcalc_main.py", line 11, in <module>
    import pandas as pd
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_102b/x86_64-centos8-gcc11-opt/lib/python3.9/site-packages/pandas/__init__.py", line 17, in <module>
    "Unable to import required dependencies:\n" + "\n".join(missing_dependencies)
ImportError: Unable to import required dependencies:
numpy:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/cvmfs/cms-bril.cern.ch/releases/brilconda-3.2.16/bin/python"
  * The NumPy version is: "1.22.3"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

failed (took 0.81 seconds)
ERROR: [pid 1525350] Worker Worker(salt=6439333770, workers=1, host=lxplus800.cern.ch, username=tomei, pid=1524837) failed    hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
Traceback (most recent call last):
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 386, in call
    return fn(task, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/software/lib/python3.9/site-packages/law/decorator.py", line 519, in localize
    return fn(task, *args, **kwargs)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/hltp/tasks.py", line 101, in run
    out = self.call_step(cmd, "running brilcalc lumi ...", publish_cmd=True)
  File "/afs/cern.ch/user/t/tomei/work/HEP/CMS/TSG/cms-hlt-parser/hltp/base.py", line 88, in call_step
    raise Exception("{} failed".format(prog or cmd.split(" ", 1)[0]))
Exception: brilcalc failed
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   FAILED
INFO: This progress looks :( because there were failed tasks

=============================== leaving sandbox ================================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================
riga commented 1 year ago

Can you try again? I just pushed another small adjustment to avoid some undesired interplay between the conda env of bril and outer python envs.

There will be some warnings such as Error in sitecustomize; set PYTHONVERBOSE for traceback that are due to some setup issues of the bril conda env itself, but brilcalc should work.

trtomei commented 1 year ago

I tried again, now it worked!

=============================== entering sandbox ===============================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

Processing /afs/cern.ch/user/t/tomei/.cache/pip/wheels/a6/7b/3c/7ae8c0b2afc734f58530d6e9239d9af303cd1cd625217adb7b/brilws-3.7.4-py3-none-any.whl
Installing collected packages: brilws
Successfully installed brilws
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
INFO: [pid 1557917] Worker Worker(salt=7457556000, workers=1, host=lxplus804.cern.ch, username=tomei, pid=1557738) running   hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
running brilcalc lumi ...
cmd: brilcalc lumi -u /pb -i /afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt --normtag /cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json --hltpath "HLT_IsoMu2?_v*"
done (took 1 minute, 48.28 seconds)
print summary of task hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)

runs : 478
lumi : 59735.969 /pb
paths: 3
  - HLT_IsoMu27_v14
  - HLT_IsoMu27_v15
  - HLT_IsoMu27_v16

INFO: [pid 1557917] Worker Worker(salt=7457556000, workers=1, host=lxplus804.cern.ch, username=tomei, pid=1557738) done      hltp.GetLumiData(hlt_path=HLT_IsoMu2?_v*, lumi_file=/afs/cern.ch/cms/CAF/CMSCOMM/COMM_DQM/certification/Collisions18/13TeV/ReReco/Cert_314472-325175_13TeV_17SeptEarlyReReco2018ABC_PromptEraD_Collisions18_JSON.txt, normtag_file=/cvmfs/cms-bril.cern.ch/cms-lumi-pog/Normtags/normtag_BRIL.json)
INFO: Informed scheduler that task   hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a   has status   DONE
INFO: This progress looks :) because there were no failed tasks or missing dependencies

=============================== leaving sandbox ================================
task   : hltp.GetLumiData_HLT_IsoMu2__v___afs_cern_ch_cms__cvmfs_cms_bril__027da4ba3a
sandbox: bash::$HLTP_BASE/sandboxes/env_bril.sh
================================================================================

Excellent, now I will continue trying to learn it. Thanks for this tool, Marcel!

Cheers, Thiago