Bug Report: Problems generating full list of scan dates for plotTimeSeries.py

ElizabethRoseStarling commented 6 years ago

Types of issue

[X] Bug report (report an issue with the code)
[ ] Feature request (request for change which adds functionality)

Expected Behavior

When I type the following command:

python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

for each detector defined in chamber_config.values() the listOfScanDAtes.txt file should be found at $DATA_PATH//scurve/

Current Behavior

Instead, it halts the process and outputs the following error:

Traceback (most recent call last): File "plotTimeSeries.py", line 26, in from gempython.utils.wrappers import envCheck ImportError: No module named gempython.utils.wrappers

Steps to Reproduce (for bugs)

python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

Possible Solution (for bugs)

Context (for feature requests)

I am trying to generate the full list of scan dates since 2017 installation for the various detectors to study the presence of burned VFAT inputs in time-series SCurve plots.

Your Environment

Version used:
Shell used: Mac terminal

mexanick commented 6 years ago

@QuantumKittydynamics, please report:

PC where you tried to run it
Account from which you've tried to run
How the package was installed (system-wide install of RPM or local github clone?)
Version of the package if installed as RPM or github branch/commit

bdorney commented 6 years ago

Additionally please include the commands, from the very beginning, and their outputs.

bdorney commented 6 years ago

Possibly related to:

https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/94 https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/95

ElizabethRoseStarling commented 6 years ago

My MacBook Pro
estarlin on LXPlus
Local github clone
commit dd3c00f47844ac620279ac57e6dd1339d188583b (I think)

mexanick commented 6 years ago

If you connect to a certain PC over ssh in your macbook terminal and then run the code there - it is that PC of our interest. So as far as I can understand, a correct answer to the first question should be ''lxplus" Now if you are using the local github clone of latest, please cross check the commit by running git show Then if you have indicated the commit correctly, you must build a package and install it under the virtual environment (this is valid for lxplus, on other PCs, like gem904* or P5 ones there could be other options)

bdorney commented 6 years ago

Then if you have indicated the commit correctly, you must build a package and install it under the virtual environment (this is valid for lxplus, on other PCs, like gem904* or P5 ones there could be other options)

Indeed I think this is why it's important to have the exact commands executed, and their output. The info above does not show this.

Also I suspect the issue is stemming from the fact that after the rebase & merge of the packaging PR's the installation and setup statements of the README.md still reflect the pre-packaging PR instructions:

https://github.com/cms-gem-daq-project/gem-plotting-tools/#setup

@QuantumKittydynamics please provide exact commands, and their output.

@BenjaminRS, @jsturdy and @mexanick is there an updated installation recipe's documented somewhere?

ElizabethRoseStarling commented 6 years ago

I see, I didn't understand you wanted the commands from genesis.

Input: ssh estarlin@lxplus.cern.ch

Output: Warning: Permanently added the RSA host key for IP address '188.185.78.215' to the list of known hosts. Password:

Input: hunter2 (No, not really)

Output: Warning: No xauth data; using fake authentication data for X11 forwarding.

Welcome to lxplus086.cern.ch, SLC, 6.9
Archive of news is available in /etc/motd-archive
Reminder: you have agreed to the CERN
computing rules, in particular OC5. CERN implements
the measures necessary to ensure compliance.
https://cern.ch/ComputingRules
Puppet environment: qa, Roger state: production
Foreman hostgroup: lxplus/nodes/login
Availability zone: cern-geneva-a
LXPLUS Public Login Service - https://cern.ch/lxplusdoc
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Input: (***) cd $BUILD_HOME/gem-plotting-tools/macros

(Because this is the only place where I can even run the script, rather than getting the following error: python: can't open file 'plotTimeSeries.py': [Errno 2] No such file or directory)

Output: It does the thing.

Input: python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

Output:

Traceback (most recent call last): File "plotTimeSeries.py", line 27, in from gempython.gemplotting.utils.anautilities import getDirByAnaType, makeListOfScanDatesFile ImportError: No module named gemplotting.utils.anautilities

*** I rebuilt the three shell variables and resourced the paths as described here: https://github.com/cms-gem-daq-project/gem-plotting-tools/#setup Hence the new error.

And, indeed, with cmsgemos on commit eda69b0c2f4f261958dc4d204943a649fc45b665 , I do not see anything called anautilities in $BUILD_HOME/cmsgemos/gempython/gemplotting/utils/anautilities

BenjaminRS commented 6 years ago

ok I will leave it to @jsturdy to give the instructions as I will likely get it wrong. 2.5 weeks ago I had to install some updates with

cp ~sturdy/public/cmsgemos_gempython-0.3.1.tar.gz .
pip install --no-deps cmsgemos_gempython-0.3.1.tar.gz

But as I said, it is best to let Jared give the correct and up-to-date instructions,

From my area on lxplus I get the following:

$ python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01
Options: vt1bump=vt1bump0, dataPath=/afs/cern.ch/user/b/benjamin/GEMDev, anaType=scurve
Traceback (most recent call last):
  File "plotTimeSeries.py", line 63, in <module>
    makeListOfScanDatesFile(chamber, anaType, options.startDate, options.endDate)
  File "/afs/cern.ch/user/b/benjamin/GEMDev/venv/slc6/py27/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 352, in makeListOfScanDatesFile
    dirPath = getDirByAnaType(anaType, chamberName, ztrim)
  File "/afs/cern.ch/user/b/benjamin/GEMDev/venv/slc6/py27/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 57, in getDirByAnaType
    from gempython.gemplotting.anaInfo import ana_config
ImportError: No module named anaInfo

jsturdy commented 6 years ago

I will try to update with a minimal get-it-working set of instructions for a user (@QuantumKittydynamics, your current use case is as a user, rather than developing a feature, correct?)

ElizabethRoseStarling commented 6 years ago

@jsturdy , yes, I'm not trying to develop anything right now, just a user. :) Thank you!

jsturdy commented 6 years ago

Hi @QuantumKittydynamics, as a first go, can you try this Missing is the cmsgemos tarball, which you can currently find on the 904 DAQ PCs in the location specified by @BenjaminRS above.

I think this will be enough get you going, but I may have missed a step or two, in which case, report here the errors.

Once this document is updated to a level that works, I'll work it into the READMEs for the different repositories

ElizabethRoseStarling commented 6 years ago

Hi @jsturdy , sorry for the late reply, I got held up by another task. So, I was able to successfully get all three tarballs, but I fail when trying to set up the virtual environment.

I checked my Python version using:

Input: python -V

Output: Python 2.7.11

So I inputted that into the instructions here: https://gist.github.com/jsturdy/a82a6b7b7d497ae8ef98ee736119c6b5

Input: [estarlin@lxplus027 CMS_GEM_DAQ]$ virtualenv -p 2.7.11 --system-site-packages /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate

Output: The executable 2.7.11 (from --python=2.7.11) does not exist

I'm not sure I understand what's happening? Thanks for your help.

lmoureaux commented 6 years ago

Hi @QuantumKittydynamics,

virtualenv expects you to provide a callable python command. If you have Python 2.7 in your PATH, you can use: virtualenv -p python ... or virtualenv -p python2.7 ...

I on my side haven't been able to find the third tarball. Is it in a "public" location somewhere?

ElizabethRoseStarling commented 6 years ago

Benjamin posted the command for the third tarball, @lmoureaux : cp ~sturdy/public/cmsgemos_gempython-0.3.1.tar.gz .

ElizabethRoseStarling commented 6 years ago

Thanks, Louis, that seemed to work.

Input:

virtualenv -p python --system-site-packages /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv

Output:

Running virtualenv with interpreter /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/python New python executable in /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/python Installing setuptools, pip...done.

This makes my venv with folders: bin include lib lib64 slc6

But the second line of the instructions doesn't make sense: /path/to/your/venv/bin/activate

I can't add it to the initial command, because I get the error "There must be only one argument: DEST_DIR"

And I can't do anything with it on its own.

Just entering it alone does nothing, as expected:

Input:

[estarlin@lxplus027 CMS_GEM_DAQ]$ /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate

Output:

-bash: /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate: Permission denied

I can't cd to it, because it doesn't exist.

And replacing the DEST_DIR with /path/to/your/venv/bin/activate results in a giant mess:

Running virtualenv with interpreter /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/python Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/virtualenv.py", line 2352, in main() File "/usr/lib/python2.6/site-packages/virtualenv.py", line 825, in main symlink=options.symlink) File "/usr/lib/python2.6/site-packages/virtualenv.py", line 985, in create_environment site_packages=site_packages, clear=clear, symlink=symlink)) File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1159, in install_python mkdir(lib_dir) File "/usr/lib/python2.6/site-packages/virtualenv.py", line 448, in mkdir os.makedirs(path) File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/../../../../../../external/python/2.7.11-ikhhed/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/../../../../../../external/python/2.7.11-ikhhed/lib/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 20] Not a directory: '/afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate/lib'

lmoureaux commented 6 years ago

Just entering it alone does nothing, as expected: Input:

[estarlin@lxplus027 CMS_GEM_DAQ]$ /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate

Output:

-bash: /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate: Permission denied

I can't cd to it, because it doesn't exist.

Source it:

source /path/to/your/venv/bin/activate

ElizabethRoseStarling commented 6 years ago

I can't source it...because it doesn't exist. Am I supposed to make it first?

Input:

[estarlin@lxplus027 CMS_GEM_DAQ]$ source /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate/

Output:

-bash: /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate/: Not a directory

lmoureaux commented 6 years ago

The error message doesn't say "No such file or directory" but "Not a directory". It believes that you're trying to access a directory. That's because of the trailing slash after activate. Remove it and it should work.

ElizabethRoseStarling commented 6 years ago

Thank you so much for all the help, Louis. I am not super command line literate yet..

So the only one I can't successfully install now is the cmsgemos tarball. Using the option --nodeps gives "no such option: --nodeps". And not adding that option gives:

Processing ./cmsgemos_gempython-0.3.1.tar.gz Collecting cactusboards-amc13-python>=1.2.5 (from cmsgemos-gempython==0.3.1) Could not find a version that satisfies the requirement cactusboards-amc13-python>=1.2.5 (from cmsgemos-gempython==0.3.1) (from versions: ) No matching distribution found for cactusboards-amc13-python>=1.2.5 (from cmsgemos-gempython==0.3.1)

lmoureaux commented 6 years ago

The correct option is --no-deps.

ElizabethRoseStarling commented 6 years ago

Great, that worked. I went through everything else, it all worked...and then finally back to the original problem...

Input:

(venv)[estarlin@lxplus027 macros]$ python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

Output:

You must source the environment properly, DATA_PATH is not set Traceback (most recent call last): File "plotTimeSeries.py", line 54, in envCheck('DATA_PATH') File "/afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/lib/python2.7/site-packages/gempython/utils/wrappers.py", line 38, in envCheck raise EnvironmentError('%s is not set'%envVar) EnvironmentError: DATA_PATH is not set

bdorney commented 6 years ago

EnvironmentError: DATA_PATH is not set

This means that shell variable $DATA_PATH is not set. However just to cross-check there's no misunderstanding what are you trying to do right now with this command? Have you already transferred the data from P5? If so this would have already generated a listOfScanDates for you.

ElizabethRoseStarling commented 6 years ago

I'm literally just trying to make the rest of the time series plot ROOT files for GEMINI 27, 28, and 30 for the sustained operations task "Re-Analysis of S-Curve Data for GEMINIm27, 28, 29, 30 between now and Installation in 2017". Your link you provided me some time ago in Mattermost is invalid, so I'm trying to follow the instructions in the /gem-plotting-tools readme, and it starts out with needing that list...

ElizabethRoseStarling commented 6 years ago

Also, in trying to fix the above problem I posted, I'm pretty sure I borked everything up again.

Louis, would you perhaps have some time tomorrow to help me, so it'll go a bit faster than trying to go step-by-step here?

bdorney commented 6 years ago

so I'm trying to follow the instructions in the /gem-plotting-tools readme, and it starts out with needing that list...

Double check the instructions. The step which generates the scandate list should be done from the P5 machine as mentioned here:

https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/develop/README.md#full-example-for-p5-s-curve-data

You'll need to perform the same steps to setup your computing area on your p5 area as you have done on lxplus.

After getting the data from the P5 machine and transferring it to lxplus I would wait until https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/95 has been resolved before trying to launch the cluster analysis. At which time we hope to update the instructions.

Unfortunately since the original instructions were provided the setup of the computing env has changed due to a parallel software development performed for the packaging projects (which was mentioned as an upcoming issue at the time). We are trying to update these extra tools (such as cluster analysis) now. Hold tight we'll get there. This debugging information is very useful!

jsturdy commented 6 years ago

Oops, missed some syntax there, as the activate script must be sourced gist updated, tldr: . /path/to/venv/bin/activate

and late-to-the-party fail on not refreshing...

@QuantumKittydynamics thanks for stumbling through my quickly thrown together (and clearly half incomplete) instructions @lmoureaux thanks also for the --no-deps catch

lmoureaux commented 6 years ago

I'm trying to setup an environment in which I can run, as instructed in the README:

$ plotTimeSeries.py --startDate=2018.05.01

But the only working environment I could find relies on an ugly hack. Please find below what I tried:

Python 2.6

I can setup the virtualenv without problem, but only at P5. On lxplus, I get:

$ virtualenv -p python2.6 --system-site-packages venv26
Running virtualenv with interpreter /usr/bin/python2.6
New python executable in venv26/bin/python2.6
Also creating executable in venv26/bin/python
Installing setuptools, pip...
Complete output from command /afs/cern.ch/user/l/...venv26/bin/python2.6 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
Traceback (most recent call last):
File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'main'
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 2352, in <module>
main()
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 825, in main
symlink=options.symlink)
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 993, in create_environment
install_wheel(to_install, py_executable, search_dirs)
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 961, in install_wheel
'PIP_NO_INDEX': '1'
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 903, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /afs/cern.ch/user/l/...venv26/bin/python2.6 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

At P5, I can't upgrade setuptools using the package downloaded from PyPI:


$  $  pip install setuptools-39.2.0.zip
Unpacking ./setuptools-39.2.0.zip
Running setup.py egg_info for package from file:///cmsnfshome0/nfshome0/lmoureau/setuptools-39.2.0.zip
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "pkg_resources/__init__.py", line 947, in <module>
    class Environment(object):
  File "pkg_resources/__init__.py", line 951, in Environment
    self, search_path=None, platform=get_supported_platform(),
  File "pkg_resources/__init__.py", line 180, in get_supported_platform
    plat = get_build_platform()
  File "pkg_resources/__init__.py", line 380, in get_build_platform
    from sysconfig import get_platform
ImportError: No module named sysconfig
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

[snipped]

from sysconfig import get_platform

ImportError: No module named sysconfig

Cleaning up... Command python setup.py egg_info failed with error code 1 in /tmp/pip-8j169k-build Storing complete log in /nfshome0/lmoureau/.pip/pip.log


In Python 2.6, `sysconfig` was part of `distutils`.

## Python 2.7

* I'm using Python 2.7 from `/opt/rh`:

source /opt/rh/python27/enable export PYTHONDIR=/opt/rh/python27/root/usr


* In this environment, the `pip` business goes more or less fine (except that I had to upload packages to P5 by hand).
* At P5, `PyROOT` is only installed for Python 2.6. I added the installed version to the `PYTHONPATH`: `export PYTHONPATH=$PYTHONPATH:/usr/lib64/python2.6/site-packages/`. *This isn't meant to be more than a temporary hack.*
* I had to `export DATA_PATH=/gemdata/` and `export ELOG_PATH=~/elog` (not a big deal but ought to be fixed)
* I was then able to run:

plotTimeSeries.py --startDate=2018.05.28



It seemed to enter an infinite loop, overwriting the same files again and again. I killed it and will investigate tomorrow.

bdorney commented 6 years ago

Hi @lmoureaux rather than re-invent the wheel could you try the official instructions for setting up a user area provided by @jsturdy? Note there is a different set of instructions for developers.

lmoureaux commented 6 years ago

Hi @lmoureaux rather than re-invent the wheel could you try the official instructions for setting up a user area provided by @jsturdy?

That's exactly what I've been trying to do

Note there is a different set of instructions for developers.

On the same page? Else can you please provide a link to it?

jsturdy commented 6 years ago

@lmoureaux did this fail at the virtualenv creation? Or when updating setuptools/pip/importlib?

I just today was trying to work around the slc6 python2.6 issues installing the updates I just tried on lxplus with the modifications I made to the gist, i.e., pip install -I importlib 'setuptools<39.2.0' 'pip<10.1.0' and the virtualenv created fine (I didn't test beyond this)

Also, on lxplus can you try a python version provided in one of the LCG package bundles? This will provide an environment with python2.7 and some version of ROOT

/cvmfs/sft.cern.ch/lcg/views/
- e.g., source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh

On the same page? Else can you please provide a link to it?

A separate gist, but it right now is mostly just a clone of the user setup, and will mostly only diverge once the user setup is stable.

lmoureaux commented 6 years ago

@jsturdy Thank you :+1:. I'll take a look tomorrow. I hadn't refreshed your gist...

lmoureaux commented 6 years ago

2.6

To update pip, importlib and setuptools in my venv at P5, I downloaded the packages on lxplus:

pip download importlib 'setuptools<39.2.0'  'pip<10.1.0'

I uploaded the tarballs using scp. When I try to install them at P5, it breaks my pip:

[lmoureau@srv-s2g18-33-01 ~]$ rm -rf venv26                                     
[lmoureau@srv-s2g18-33-01 ~]$ virtualenv -p python --system-site-packages venv26
Already using interpreter /usr/bin/python
New python executable in venv26/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip..........................................................................................................................................................................................................................................................................................................................................done.
[lmoureau@srv-s2g18-33-01 ~]$ source venv26/bin/activate
(venv26)[lmoureau@srv-s2g18-33-01 ~]$ pip install -U importlib-1.0.4.zip pip-9.0.3-py2.py3-none-any.whl setuptools-36.8.0-py2.py3-none-any.whl 
Unpacking ./importlib-1.0.4.zip
  Running setup.py egg_info for package from file:///cmsnfshome0/nfshome0/lmoureau/importlib-1.0.4.zip
Unpacking ./pip-9.0.3-py2.py3-none-any.whl
Unpacking ./setuptools-36.8.0-py2.py3-none-any.whl
Installing collected packages: importlib, pip, setuptools
  Running setup.py install for importlib
  Found existing installation: pip 1.4.1
    Uninstalling pip:
      Successfully uninstalled pip
  Found existing installation: distribute 0.6.10
    Not uninstalling distribute at /usr/lib/python2.6/site-packages, outside environment /cmsnfshome0/nfshome0/lmoureau/venv26
Successfully installed importlib pip setuptools
Cleaning up...
(venv26)[lmoureau@srv-s2g18-33-01 ~]$ pip
-bash: /cmsnfshome0/nfshome0/lmoureau/venv26/bin/pip: No such file or directory

Note: The system pip is also broken:

$ /usr/bin/pip --version
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.3.1', 'console_scripts', 'pip')()
[snipped]
  File "/usr/lib64/python2.6/io.py", line 631, in __init__
    _fileio._FileIO.__init__(self, name, mode, closefd)
IOError: [Errno 13] Permission denied: '/usr/lib/python2.6/site-packages/cactusboards_amc13_python-1.2.5-py2.6.egg-info'

2.7

Also, on lxplus can you try a python version provided in one of the LCG package bundles? This will provide an environment with python2.7 and some version of ROOT

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ python -V
Python 2.7.13
$ virtualenv -p python --system-site-packages venv27
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 2, in <module>
    import virtualenv
ImportError: No module named virtualenv

I get the same results even after pip install --user virtualenv.

jsturdy commented 6 years ago

OK, the system pip issue is not pip per se, just some bad permissions on a package I built and installed (I had fixed it on gem-dqm01, but forgot to fix it on gem-daq01, so that is now fixed)

OK, on the LCG package, this is because virtualenv must not be included in the set..., might be worth asking on the computing tools hypernews. Actually, i just tried the following:

source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
python -m pip list
python -m pip install -U importlib setuptools pip

This worked, probably due to the way that the LCG bundle is packaged, and in general, it should always work, even if the pip and virtualenv binary method works.

bdorney commented 6 years ago

For setting up a virtualenv on lxplus i think the example illustrated here:

https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/develop/setup/createVirtualEnvPy27.sh

Could be adapted, note there is one for py26 as well.

Again I would suggest against trying to reinvent the wheel and try to adapt existing tools.

jsturdy commented 6 years ago

That will only work if the rh-python27 package is installed on the machine (I have no idea what the CERN policy is on that), the LCG views are a "standard" option available to any lxplus user

One can also get a cc7 machine by using lxplus7 and get python2.7 as the default, which can also be more stable than relying on an LCG view, or a possibly installed extra.

In addition, it's not clear that that method would work with the updates to the three packages (I'm pretty sure the same issues already encountered would be present), as a "standard" virtualenv setup is not a problem, it's a problem if there are some user installed local python packages (pip list --user) or some compatibility issues between the versions of these updates, which I previously had not placed upper limits on, but probably should, simply because python2.6 is likely not tested, and python2.7 is EOL in 2020.

bdorney commented 6 years ago

That will only work if the rh-python27 package is installed on the machine (I have no idea what the CERN policy is on that), the LCG views are a "standard" option available to any lxplus user

Indeed, but it seems lxplus nodes have rh-python27 installed. However the point is taken. The script could be updated to use the LCG if that was desired.

One can also get a cc7 machine by using lxplus7 and get python2.7 as the default, which can also be more stable than relying on an LCG view, or a possibly installed extra.

Does the subversion of python2.7 matter much here? I assume not, on cc7 the default is 2.7.5 whereas rh-python27 is 2.7.11 iirc.

In addition, it's not clear that that method would work with the updates to the three packages (I'm pretty sure the same issues already encountered would be present), as a "standard" virtualenv setup is not a problem, it's a problem if there are some user installed local python packages (pip list --user) or some compatibility issues between the versions of these updates, which I previously had not placed upper limits on, but probably should, simply because python2.6 is likely not tested, and python2.7 is EOL in 2020.

I'll give it a try as well and report back. I'm trying the developer instructions so @lmoureaux can continue with the user instructions.

lmoureaux commented 6 years ago

Python 2.7

On my lxplus account:

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
/cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python: No module named virtualenv

However,

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ pip install --user virtualenv
$ python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
Already using interpreter /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in /tmp/lmoureau/lcgvenv/bin/python
Installing setuptools, pip, wheel...done.

...and the rest of the instructions work fine, though:exclamation:the gemplotting macros are not executable.

We may want to use $VIRTUAL_ENV in the instructions.

What's a good Python 2.7 distribution available at P5?

bdorney commented 6 years ago

...and the rest of the instructions work fine, though❗️the gemplotting macros are not executable.

What is the output of echo $PATH?

We may want to use $VIRTUAL_ENV in the instructions.

What do you mean?

lmoureaux commented 6 years ago

Python 2.6

Still on my lxplus account, using the default Python :

python -m pip throws an exception, one has to use pip
Else (on lxplus) everything is as for Python 2.7

lmoureaux commented 6 years ago

the gemplotting macros are not executable.

What is the output of echo $PATH?

The PATH is set correctly, but one has to chmod +x the macros. This is a problem in the package.

We may want to use $VIRTUAL_ENV in the instructions.

What do you mean?

Instead of:

export PATH=<path/to/venv/lib/../site-packages/gempython/scripts>:$PATH

write:

export "PATH=$VIRTUAL_ENV/lib/python*/site-packages/gempython/scripts:$PATH"

which can be copy-pasted as-is, provided there's only one Python version installed in the virtualenv. It can be changed in two other locations.

I also think that the .. part of the path should be replaced, as it has a meaning in this context.

bdorney commented 6 years ago

I also think that the .. part of the path should be replaced, as it has a meaning in this context.

I think @jsturdy was using it simply to imply additional sub folders exist, and includes a **do not blindly copy this!** warning. However if instead of using VIRTUAL_ENV I would suggest to use instead VENV_ROOT following the same schema we have used in the past for our other repo's (e.g. shell variables defined XHAL_ROOT and CTP7_ROOT.

Additionally in:

export "PATH=$VENV_ROOT/lib/python*/site-packages/gempython/scripts:$PATH"

The number of python versions installed should not matter since you should be setting up a separate virtualenv with a different VENV_ROOT for a different python directory, the lib/python* is already inside the venv.

lmoureaux commented 6 years ago

I also think that the .. part of the path should be replaced, as it has a meaning in this context.

I think @jsturdy was using it simply to imply additional sub folders exist

That's obviously the meaning of it, but I didn't understand it at first (there are two parts to replace). Just trying to avoid complaints about the instructions not working

instead of using VIRTUAL_ENV I would suggest to use instead VENV_ROOT

My point is that source foo/bar/bin/activate already sets VIRTUAL_ENV to the correct value for virtualenv currently in use.

provided there's only one Python version installed in the virtualenv

The number of python versions installed should not matter since you should be setting up a separate virtualenv with a different VENV_ROOT for a different python directory, the lib/python* is already inside the venv.

Even though I couldn't find one, maybe @jsturdy had a good reason not to use a wildcard here.

bdorney commented 6 years ago

My point is that source foo/bar/bin/activate already sets VIRTUAL_ENV to the correct value for virtualenv currently in use.

I see, I envisioned a different use case in which $VENV_ROOT is an argument provided by the user which is used in a script to create the venv in the correct location.

Even though I couldn't find one, maybe @jsturdy had a good reason not to use a wildcard here.

When using wildcards the following fails for me:

find <path/to/venv/lib/../site-packages/gempython> -type f -name buildMapFiles.py -exec python {} \;

In general I would suggest PATH to not include wildcards. You can do something like:

PYTHON_VERSION=$(python -c "import sys; sys.stdout.write(sys.version[:3])") 
export "PATH=$VENV_ROOT/lib/python$PYTHON_VERSION/site-packages/gempython/scripts:$PATH"

With some help from @jsturdy got the syntax working nicely.

lmoureaux commented 6 years ago

When using wildcards the following fails for me:
find <path/to/venv/lib/../site-packages/gempython> -type f -name buildMapFiles.py -exec python {} \;

How does it fail? I just checked and for me it finds the file (not sure about executing it).

In general I would suggest PATH to not include wildcards.

Wildcards are expanded when the variable is set, and AFAIK not when looking it up. So my command above $PATH will only contain the wildcard when it didn't match anything, and even in that case it will be harmless. We can construct it from the Python version, although a cleaner (in the long term) option would be to install the tools in venv/bin, so that they are in the PATH to begin with :-)

bdorney commented 6 years ago

source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh

@jsturdy this probably also covers devtoolset-6/7 I assume as well?

bdorney commented 6 years ago

I cannot repeat @lmoureaux's success story:

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh $ pip install --user virtualenv $ python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages Already using interpreter /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python PYTHONHOME is set. You must activate the virtualenv before using it New python executable in /tmp/lmoureau/lcgvenv/bin/python Installing setuptools, pip, wheel...done.

specifically:

...and the rest of the instructions work fine

When attempting to run pip install -U importlib setuptools pip:

Running virtualenv with interpreter /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in /afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/slc6/v2b/py27/bin/python
Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/slc6/v2b/py27/bin/pip", line 7, in <module>
    from pip._internal import main
ImportError: No module named _internal

With the LCG version of pip and python there seems to be some anomaly in pip after creating the user installed virtualenv. The internet suggests this is because there are multiple versions of pip in $PATH and the attempt to upgrade is ambiguous; a bit surprised by this actually. Seems like LCG is leaving some residual things in $PATH that are ambiguous (or PYTHONPATH is not set correctly...?).

Going to give up on the LCG view since this doesn't seem to be functional. Diverting back to the relying on the original strategy shown in createVirtualEnvPy27.sh

lmoureaux commented 6 years ago

When attempting to run pip install -U importlib setuptools pip:

At the time, the instructions stated:

python -m pip install -U importlib setuptools pip

jsturdy commented 6 years ago

@lmoureaux under which conditions was python -m pip working and when did pip alone work? I had modified the instructions (with a note that if pip/virtualenv themselves fail, to try with python -m pip/virtualenv, based on your reported semi-success)

jsturdy commented 6 years ago

@bdorney

source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh

@jsturdy this probably also covers devtoolset-6/7 I assume as well?

I definitely don't know what's being asked here... if a person is developing a feature that depends on some special version of some tool, I fully expect them to be able to set this up themselves (or follow a README/CONTRIBUTING that tells them what to do). The current attempt at providing a set of instructions is specific to the python side of things, and makes no attempt at wider support (which could be added to the aforementioned README/CONTRIBUTING in the repository of interest)

lmoureaux commented 6 years ago

@lmoureaux under which conditions was python -m pip working and when did pip alone work?

The ones documented here. It's maybe due to me having installed virtualenv locally.

cms-gem-daq-project / gem-plotting-tools