matthewfeickert
commented
3 years ago Thanks for the Issue @jburzy01. What you are describing is an issue with how HEP software abuses the PYTHONPATH environment variable.
If you just do something on LXPLUS like setup a reasonable LCG view for Python 3
$ unset PYTHONPATH
$ lsetup 'views LCG_98python3 x86_64-centos7-gcc8-opt'
************************************************************************
Requested: views ...
Setting up views LCG_98python3:x86_64-centos7-gcc8-opt ...
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
************************************************************************
$ echo $PYTHONPATH
/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packagesyou see that you already have a problem because LCG is now going to have control over site-packages and so will pollute your environment, making your virtual environment no longer guaranteed to be safe.
If for example you safeguard your virtual environment at login with something like the following in your .bashrc
# Setup default LCG view
default_LCG_release="LCG_98python3"
default_LCG_platform="x86_64-centos7-gcc8-opt"
printf "\nlsetup 'views ${default_LCG_release} ${default_LCG_platform}'\n"
lsetup "views ${default_LCG_release} ${default_LCG_platform}"
# strip out LGG view site-packages from PYTHONPATH to allow venv control of pip
IFS=':' read -ra _LCG_paths <<< "${PYTHONPATH}"
for idx in "${!_LCG_paths[@]}"
do
[[ "${_LCG_paths[$idx]}" == *"site-packages"* ]] && unset -v "_LCG_paths[${idx}]"
done
export PYTHONPATH=$(IFS=:; echo "${_LCG_paths[*]}")then your default login will now give your virtual environments full control over themselves (note no site-packages in PYTHONPATH now)
$ echo $PYTHONPATH
/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/libAs an explicit example, if we start with this sort of .bashrc guarding and then make a new virtual environment, everything is fine
$ cd $HOME
$ python -m venv example
(example) $ . example/bin/activate
(example) $ echo $PYTHONPATH
/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib
(example) $ python -m pip install --quiet --upgrade pip setuptools wheel
(example) $ python -m pip install --quiet pandamonium
(example) $ pip show pandamonium # We see that pandamonium lives in our virtual environment's site-packages
Name: pandamonium
Version: 0.2.1
Summary: Command line library to parse the Panda web API
Home-page: https://github.com/dguest/pandamonium
Author: Dan Guest, Matthew Feickert
Author-email: daniel.hay.guest@cern.ch, matthew.feickert@cern.ch
License: BSD 3-Clause
Location: /home/feickert/example/lib/python3.7/site-packages
Requires: panda-client
Required-by:
(example) $ pip list
Package Version
------------ -------
panda-client 1.4.45
pandamonium 0.2.1
pip 20.3.3
pyspark 2.4.6
setuptools 51.1.1
wheel 0.36.2
(example) $ python -m pip list | wc -l
8However, if we just setup a new LCG view it will prepend /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages to PYTHONPATH which means that is the first site-packages that will be found by your shell and so your virtual environment is effectively destroyed, as LCG views are forms of virtual environments of their own.
(example) $ lsetup 'views LCG_98python3 x86_64-centos7-gcc8-opt'
************************************************************************
Requested: views ...
Setting up views LCG_98python3:x86_64-centos7-gcc8-opt ...
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
************************************************************************
(example) $ echo $PYTHONPATH
/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib
(example) $ python -m pip list | grep pandamonium
(example) $ python -m pip list | wc -l # Note this was previously 8
280Anytime you setup an ATLAS software release you will be doing the same sort of thing as it is setting up specific views that trash PYTHONPATH. In general, this is totally horrible and should never be done, but HEP has decided this is how we are going to do things.
To get access to your installed libraries in your virtual environment you'll need to update the shell environment again by deactivating and activating the virtual environment again, but as you can see the LCG view is still there polluting it.
(example) $ deactivate
$ . example/bin/activate
(example) $ python -m pip list | grep pandamonium
pandamonium 0.2.1
(example) $ python -m pip list | wc -l
282I realize that so far I've just reshown you your problem. In general, there is no "solution" to your problem as if you are trying to use an interactive shell you need to choose one and only one environment that you will work in for that shell. If you want to be able to fluidly switch between environments and LCG views then I would recommend (what I do) exclusively running all ATLAS code in subshells so that is can never pollute my interactive environment (so never source anything in your interactive shell but just write a short script and then bash your_shell_script.sh).
If this is not reasonable for you then I think you might need to install pandamonium like @dguest does by cloning the repo and then putting the shell scripts in PATH, which should be a bit more safe from the radical shell environment changes that ATLAS software imposes.
I'm not sure if this is helpful at all, so please let us know if you have follow up questions here.
dguest
dantrim
jburzy01
I'm running pandamon on lxplus via the virtualenv described in the README. It works fine out of the box, but if I have set up an AnalysisBase release I get
Has anyone else encountered this or have any thoughts on how to fix it?