Upgrade to Coffea 0.7.22

[alejands]

Based on discussion in https://github.com/mcremone/decaf/pull/77.

Update packages necessary to run on Coffea version 0.7.22

[mcremone]

@alejands were you able to make this work upgrading all the other packages? To be super sure would be good to test the following scripts:

macros/pack.py utils/common.py utils/corrections.py (the version @ParticleChef is working on) utils/ids.py (the version you put in your PR)

[alejands]

@mcremone I'm not sure I understand the question. So far I've only upgraded coffea to 0.7.20 in PR https://github.com/mcremone/decaf/pull/77. I haven't made any other changes. The only file I had tested was ids.py, which ran successfully.

I'll be sure to test the other files you mentioned.

[mcremone]

@alejands I was referring to upgrading the versions of uproot and python to be able to use 0.7.22 instead of 0.7.20, as you were describing here:

https://github.com/mcremone/decaf/pull/77#issuecomment-1857238813

this means changing this script:

https://github.com/mcremone/decaf/blob/UL/setup_lcg.sh

by pip installing coffea==0.7.22, python==3.7, awkward==1.10.5. Once you do that you need to retest the following scripts:

macros/pack.py utils/common.py utils/corrections.py (the version @ParticleChef is working on) utils/ids.py (the version you put in your PR)

once that is done you can make a PR to change setup_lcg.sh.

That being said, I'm surprised that, once yon install coffea==0.7.22, the right versions of uproot and python don't get automatically installed.

[alejands]

@mcremone I believe uproot does get updated accordingly when running pip install --user coffea=0.7.XX. I'll double check this.

My understanding is that pip bases what it can install on the python version it sees. It will only look for packages compatible with the current python version. Pip itself can't install python; it has to be installed/sourced separately.

[alejands]

The python version is fetched in this line

https://github.com/mcremone/decaf/blob/8172777d5475cd8acd30ceeac006f14fd38ea2fe/env_lcg.sh#L9

I see that on LPC, only Python 2.7 and 3.6 are installed in .local for each user.

[alejands@cmslpcgpu3 ~]$ ls .local/lib
python2.7  python3.6

[mcremone]

Can you check what we have on KISTI? Also, can you check if you can install different python versions in your .local folder, both at the LPC and KISTI? You should be able to.

[mcremone]

I checked and KISTI also only has python 3.6. But you can install python 3.11. I tried installing and it works. You can use the following commands:

wget https://www.python.org/ftp/python/3.11.7/Python-3.11.7.tar.xz
tar -Jxvf Python-3.11.7.tar.xz
cd Python-3.11.7/
./configure --prefix=/cms/ldap_home/user_name/.local
make
make install

[alejands]

@mcremone Sorry I meant to write back earlier. I have a flight tomorrow and I've been preparing for that and left this comment half written.

I still don't have KISTI access. I still have to email them about resetting my password. I'll try to do that soon.

I was able to find python 3.9 within the LCG libraries on LPC. Here is the LCG version I'm using: https://lcginfo.cern.ch/release_packages/104c/x86_64-centos7-gcc11-opt/

This is the brief documentation I found on the USCMS website that pointed me to this: https://uscms.org/uscms_at_work/computing/setup/setup_software.shtml#lcgsoft

I was able to install coffea 0.7.22 by making the following changes to the env_lcg.sh file:

else
  #source /cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/setup.sh
  source /cvmfs/sft.cern.ch/lcg/views/LCG_104c/x86_64-centos7-gcc11-opt/setup.sh
fi

# export PYTHONPATH=~/.local/lib/python3.6/site-packages:$PYTHONPATH
export PYTHONPATH=~/.local/lib/python3.9/site-packages:$PYTHONPATH

I can also try to install python 3.11 with your method if you want.

[mcremone]

@alejands you can access KISTI from lxplus, I pinned the message on the #decaf channel on slack and I'm also copying it here:

For people using KISTI to run decaf
@Erdem Yigit Ertorer has just registered lxplus913 and lxplus921
meaning that you can first login into one of these two lxplus nodes
and then ssh into KISTI from there
give it a try

No need to install 3.11 is 3.9 works.

[alejands]

@mcremone Yes that works for me and the password prompt pops up. The issue is that it doesn't recognize my password

No need to install 3.11 is 3.9 works.

Sounds good

[alejands]

The scripts have been updated in PR https://github.com/mcremone/decaf/pull/84.

[mcremone]

I have just learned something that can simplify the analysis workflow and it is relevant for this. Apparently combine now works with python 3:

http://cms-analysis.github.io/HiggsAnalysis-CombinedLimit/

If I read this correctly, CMSSW_11_3_4 should have now a python 3 installation. If it's a python version larger than 3.8 we can now run decaf within a cmsenv.

[mcremone]

I checked, it looks like it's 3.8, so we can use it. @alejands

[alejands]

@mcremone Do we need CMSSW_11_3_X in particular for Run2 UL, or can we move to any version for combine?

This way we could use CMSSW_13_3_X since we're already using it for DeepMET (albeit Run3).

[mcremone]

Unfortunately we should do CMSSW_11_3_4, as described in here:

http://cms-analysis.github.io/HiggsAnalysis-CombinedLimit/

you'll then place decaf inside CMSSW_11_3_4/src. To work with this you need to modify this line:

https://github.com/alejands/decaf/blob/UL_update_coffea/setup_lcg.sh#L5

into:

pip3 install --user coffea==0.7.22

Since you are there you may also modify this:

https://github.com/alejands/decaf/blob/UL_update_coffea/setup_lcg.sh#L6

to install the official rhalphalib:

https://github.com/nsmith-/rhalphalib/tree/master

modulo we propagate our changes into the official repo, which we will do.

Then you'll remove this part:

https://github.com/alejands/decaf/blob/UL_update_coffea/env_lcg.sh#L1-L7

and you change this line:

https://github.com/alejands/decaf/blob/UL_update_coffea/env_lcg.sh#L9

into:

export PYTHONPATH=~/.local/lib/python3.8/site-packages:$PYTHONPATH

Once that is done, you may want to restart testing the scripts listed at the beginning of this issue.

[alejands]

Sounds good. Thanks for the info!

[alejands]

I made the changes listed in https://github.com/mcremone/decaf/issues/83#issuecomment-1865195676 and added them to the branch in PR https://github.com/mcremone/decaf/pull/84, but I ran into a couple of errors.

Minor note: rather than use pip3 install --user ... , I used python3 -m pip install --user ... . The latter is equivalent, but is explicit as to which pip version is being used. This is also what the rhalphalib README uses.

---

During the coffea installation, I got warnings related to packages not on PATH, and errors from pip when it came to some packages that I don't think we use (I assume these come with the CMSSW Python version). However, I'm unable to recreate the errors when rerunning. They only seem to appear when running the installations for the first time.

So far I've tested running python3 utils/ids.py (my version) and python3 utils/common.py (the UL branch version) and got no errors. For now I think we can look the other way...

Details

``` Installing collected packages: numpy, mdurl, zipp, typing-extensions, pygments, markdown-it-py, awkward0, uproot3-methods, rich, llvmlite, boost-histogram, uproot3, uproot, pyarrow, numba, lz4, fsspec, correctionlib, awkward, coffea WARNING: The scripts f2py, f2py3 and f2py3.8 are installed in '/uscms/home/alejands/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script pygmentize is installed in '/uscms/home/alejands/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script markdown-it is installed in '/uscms/home/alejands/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script correction is installed in '/uscms/home/alejands/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. dxr 0.1 requires Flask>=0.9, which is not installed. dxr 0.1 requires futures>=2.1.1, which is not installed. rep 0.6.6 requires bokeh==0.11.1, but you have bokeh 2.3.0 which is incompatible. rep 0.6.6 requires hep-ml==0.4, but you have hep-ml 0.6.2 which is incompatible. rep 0.6.6 requires matplotlib==1.5.1, but you have matplotlib 3.3.4 which is incompatible. rep 0.6.6 requires mpld3==0.2, but you have mpld3 0.5.2 which is incompatible. rep 0.6.6 requires nose-parameterized==0.5.0, but you have nose-parameterized 0.6.0 which is incompatible. rep 0.6.6 requires notebook==4.2.1, but you have notebook 6.2.0 which is incompatible. rep 0.6.6 requires pandas==0.17.1, but you have pandas 1.2.2 which is incompatible. rep 0.6.6 requires pybrain==0.3, but you have pybrain 0.3.1 which is incompatible. rep 0.6.6 requires scikit-learn==0.17.1, but you have scikit-learn 0.24.1 which is incompatible. rep 0.6.6 requires theano==0.8.2, but you have theano 1.0.5 which is incompatible. rep 0.6.6 requires xgboost==0.4a30, but you have xgboost 1.3.3 which is incompatible. tensorflow 2.4.1 requires gast==0.3.3, but you have gast 0.4.0 which is incompatible. tensorflow 2.4.1 requires grpcio~=1.32.0, but you have grpcio 1.36.0 which is incompatible. tensorflow 2.4.1 requires numpy~=1.19.2, but you have numpy 1.23.5 which is incompatible. tensorflow 2.4.1 requires opt-einsum~=3.3.0, but you have opt-einsum 0+untagged.1.g2457b55 which is incompatible. tensorflow 2.4.1 requires typing-extensions~=3.7.4, but you have typing-extensions 4.9.0 which is incompatible. tensorflow 2.4.1 requires wheel~=0.35, but you have wheel 0.33.6 which is incompatible. tensorflow 2.4.1 requires wrapt~=1.12.1, but you have wrapt 1.11.2 which is incompatible. dxr 0.1 requires parsimonious==0.5, but you have parsimonious 0.8.1 which is incompatible. Successfully installed awkward awkward0 boost-histogram coffea correctionlib fsspec llvmlite lz4 markdown-it-py mdurl numba numpy pyarrow pygments rich typing-extensions uproot uproot3 uproot3-methods zipp ```

---

The line enabling Jupyter also gives an error, and this one I'm unsure about.

Details

``` Traceback (most recent call last): File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_4/external/slc7_amd64_gcc900/bin/jupyter-nbextension", line 6, in from pkg_resources import load_entry_point File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3251, in @_call_aside File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside f(*args, **kwargs) File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master return cls._build_from_requirements(__requires__) File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 791, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (pygments 2.17.2 (/uscms/homes/a/alejands/.local/lib/python3.8/site-packages), Requirement.parse('pygments<2.6'), set(['ipython'])) ```

[alejands]

The README may also need to be updated to reflect moving to a CMSSW work area. Something like,

cmsrel CMSSW_11_3_4
cd CMSSW_11_3_4/src
git clone git@github.com:mcremone/decaf.git # I clone my own fork but same thing
cd decaf
source setup_env.sh

Also note that with CMSSW_11_3_4, we need to explicitly use python3 and not python (which runs Python 2.7.15+).

[mcremone]

I made the changes listed in #83 (comment) and added them to the branch in PR #84, but I ran into a couple of errors.

Minor note: rather than use pip3 install --user ... , I used python3 -m pip install --user ... . The latter is equivalent, but is explicit as to which pip version is being used. This is also what the rhalphalib README uses.

During the coffea installation, I got warnings related to packages not on PATH, and errors from pip when it came to some packages that I don't think we use (I assume these come with the CMSSW Python version). However, I'm unable to recreate the errors when rerunning. They only seem to appear when running the installations for the first time.

So far I've tested running python3 utils/ids.py (my version) and python3 utils/common.py (the UL branch version) and got no errors. For now I think we can look the other way...

Details

Installing collected packages: numpy, mdurl, zipp, typing-extensions, pygments, markdown-it-py, awkward0, uproot3-methods, rich, llvmlite, boost-histogram, uproot3, uproot, pyarrow, numba, lz4, fsspec, correctionlib, awkward, coffea
  WARNING: The scripts f2py, f2py3 and f2py3.8 are installed in '/uscms/home/alejands/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script pygmentize is installed in '/uscms/home/alejands/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script markdown-it is installed in '/uscms/home/alejands/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script correction is installed in '/uscms/home/alejands/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dxr 0.1 requires Flask>=0.9, which is not installed.
dxr 0.1 requires futures>=2.1.1, which is not installed.
rep 0.6.6 requires bokeh==0.11.1, but you have bokeh 2.3.0 which is incompatible.
rep 0.6.6 requires hep-ml==0.4, but you have hep-ml 0.6.2 which is incompatible.
rep 0.6.6 requires matplotlib==1.5.1, but you have matplotlib 3.3.4 which is incompatible.
rep 0.6.6 requires mpld3==0.2, but you have mpld3 0.5.2 which is incompatible.
rep 0.6.6 requires nose-parameterized==0.5.0, but you have nose-parameterized 0.6.0 which is incompatible.
rep 0.6.6 requires notebook==4.2.1, but you have notebook 6.2.0 which is incompatible.
rep 0.6.6 requires pandas==0.17.1, but you have pandas 1.2.2 which is incompatible.
rep 0.6.6 requires pybrain==0.3, but you have pybrain 0.3.1 which is incompatible.
rep 0.6.6 requires scikit-learn==0.17.1, but you have scikit-learn 0.24.1 which is incompatible.
rep 0.6.6 requires theano==0.8.2, but you have theano 1.0.5 which is incompatible.
rep 0.6.6 requires xgboost==0.4a30, but you have xgboost 1.3.3 which is incompatible.
tensorflow 2.4.1 requires gast==0.3.3, but you have gast 0.4.0 which is incompatible.
tensorflow 2.4.1 requires grpcio~=1.32.0, but you have grpcio 1.36.0 which is incompatible.
tensorflow 2.4.1 requires numpy~=1.19.2, but you have numpy 1.23.5 which is incompatible.
tensorflow 2.4.1 requires opt-einsum~=3.3.0, but you have opt-einsum 0+untagged.1.g2457b55 which is incompatible.
tensorflow 2.4.1 requires typing-extensions~=3.7.4, but you have typing-extensions 4.9.0 which is incompatible.
tensorflow 2.4.1 requires wheel~=0.35, but you have wheel 0.33.6 which is incompatible.
tensorflow 2.4.1 requires wrapt~=1.12.1, but you have wrapt 1.11.2 which is incompatible.
dxr 0.1 requires parsimonious==0.5, but you have parsimonious 0.8.1 which is incompatible.
Successfully installed awkward awkward0 boost-histogram coffea correctionlib fsspec llvmlite lz4 markdown-it-py mdurl numba numpy pyarrow pygments rich typing-extensions uproot uproot3 uproot3-methods zipp

The line enabling Jupyter also gives an error, and this one I'm unsure about.

Details

Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_4/external/slc7_amd64_gcc900/bin/jupyter-nbextension", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3251, in <module>
    @_call_aside
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py2-setuptools/44.1.1/lib/python2.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pygments 2.17.2 (/uscms/homes/a/alejands/.local/lib/python3.8/site-packages), Requirement.parse('pygments<2.6'), set(['ipython']))

The error Jupyter message is a version conflict. As you can see from the printout it tries to use python 2, but it searches into the python 3 local folder. I would comment this line out for the moment, or find a way to run it with python 3.

[mcremone]

The README may also need to be updated to reflect moving to a CMSSW work area. Something like,

cmsrel CMSSW_11_3_4
cd CMSSW_11_3_4/src
git clone git@github.com:mcremone/decaf.git # I clone my own fork but same thing
cd decaf
source setup_env.sh

Also note that with CMSSW_11_3_4, we need to explicitly use python3 and not python (which runs Python 2.7.15+).

I would actually also include the full combine installation, since we are gonna use it anyway.

[alejands]

@mcremone The changes discussed have been added to PR https://github.com/mcremone/decaf/pull/84.

• 1415be5: Rename the setup scripts
• c0d4f35: Begin rewriting the README and add instructions on installing combine
• 59e93ba: Comment out the jupyter line from setup.sh

The PR should be good to go.

[alejands]

Task completed with PR #84 merged