Closed bdorney closed 6 years ago
Trying the previous hack I had:
pip uninstall --yes root-numpy
cd /tmp
git clone git://github.com/rootpy/root_numpy.git
cd root_numpy
python setup.py install
cd /data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/trim/z4.000000/2018.06.01.04.39
anaUltraScurve.py -i SCurveData_trimdac0.root -c -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Now the calibration file is successfully parsed.
I think there is an issue with installation of the package. Specifically on a gem904
machine:
mkdir -p <some/path>
python -m virtualenv -p python --system-site-packages <some/path>
. <some/path>/bin/activate
python -m pip install -U importlib setuptools pip
python -m pip install gempython_gemplotting-1.0.0.tar.gz
Does not ensure that root_numpy
is properly present.
@jsturdy looks like another issue we need to follow-up on. Removing the "bug" tag since it is not an issue with the software.
4.7.3
of root_numpy
? (seems unlikely if the git
installed method worked)gemplotting
does the output print any information about checking for and installing root_numpy
?Making a new venv
:
source $BUILD_HOME/setup_gemdaq.sh -w -c 0.3.1 -g 1.0.0 -G 5 -v 2.0.0 -V 3 -p $PWD/venv_test 2>&1 | tee venvLog.log
Then when installing gemplotting
relevant output from venvLog.log
is:
python -m pip install gempython_gemplotting-1.0.0.tar.gz
Processing ./gempython_gemplotting-1.0.0.tar.gz
Requirement already satisfied: root_numpy in /usr/lib64/python2.7/site-packages (from gempython-gemplotting==1.0.0) (4.7.3)
Requirement already satisfied: numpy<1.15,>1.7 in /usr/lib64/python2.7/site-packages (from gempython-gemplotting==1.0.0) (1.7.1)
Building wheels for collected packages: gempython-gemplotting
Running setup.py bdist_wheel for gempython-gemplotting: started
Running setup.py bdist_wheel for gempython-gemplotting: finished with status 'done'
Stored in directory: /afs/cern.ch/user/d/dorney/.cache/pip/wheels/93/1c/13/b26f22301c4d62fc111e81bc48d1946cccc94e8877b9c86daa
Successfully built gempython-gemplotting
xstatic-jquery-ui 1.12.0.1 requires XStatic-jQuery, which is not installed.
sphinxcontrib-spelling 2.3.0 requires PyEnchant>=1.6.5, which is not installed.
pyramid 1.5.6 requires WebOb>=1.3.1, which is not installed.
sphinx-rtd-theme 0.1.8 has requirement sphinx>=1.3, but you'll have sphinx 1.1.3 which is incompatible.
rtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you'll have pyudev 0.15 which is incompatible.
ipapython 4.5.4 has requirement dnspython>=1.15, but you'll have dnspython 1.12.0 which is incompatible.
Installing collected packages: gempython-gemplotting
Found existing installation: gempython-gemplotting 1.0.0
Not uninstalling gempython-gemplotting at /usr/lib/python2.7/site-packages, outside environment /home/dorney/venv_test/cc7/py2.7
Can't uninstall 'gempython-gemplotting'. No files were found to uninstall.
Successfully installed gempython-gemplotting-1.0.0
Which makes a check on root-numpy
:
Requirement already satisfied: root_numpy in /usr/lib64/python2.7/site-packages (from gempython-gemplotting==1.0.0) (4.7.3)
However issue persists:
% anaUltraScurve.py -i SomeSCurveFile.root -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Analyzing: 'SomeSCurveFile.root'
Traceback (most recent call last):
File "/home/dorney/venv_test/cc7/py2.7/lib/python2.7/site-packages/gempython/scripts/anaUltraScurve.py", line 191, in <module>
tuple_calInfo = parseCalFile(options.calFile)
File "/home/dorney/venv_test/cc7/py2.7/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 429, in parseCalFile
array_CalData = rp.tree2array(tree=calTree, branches=list_bNames)
AttributeError: 'module' object has no attribute 'tree2array'
I don't think the issue is with 4.7.3
of root-numpy
but with how the package is being installed. I am not sure it's effectively installed:
% pip uninstall --yes root-numpy
Not uninstalling root-numpy at /usr/lib64/python2.7/site-packages, outside environment /home/dorney/venv_test/cc7/py2.7
Can't uninstall 'root-numpy'. No files were found to uninstall.
Then attempting to install root-numpy
using the hack above works. Trying to run anaUltraScurve.py
works without issue:
cd /data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/trim/z4.000000/2018.06.01.04.39
anaUltraScurve.py -i SCurveData_trimdac0.root -c -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Processed no problem =/
What about
python -m pip install -I gempython_gemplotting-1.0.0.tar.gz
i.e., force reinstalling packages (though not sure about dependencies...)
Also, you have a typo:
pip uninstall --yes root-numpy
should be
pip uninstall --yes root_numpy
Installing with:
python -m pip install -I gempython_gemplotting-1.0.0.tar.gz
Has seemed to resolve the issue with rp.tree2array(...)
not being found. However it also seems that the version of root-numpy
that was installed on gem904qc8daq
was installed against root 5.43.Y
while ROOT 6.12/04
is installed on the machine:
% anaUltraScurve.py -i SCurveData_trimdac0.root -c -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Analyzing: 'SCurveData_trimdac0.root'
mkdir: cannot create directory ‘SCurveData_trimdac0’: File exists
Traceback (most recent call last):
File "/home/dorney/venv_test/cc7/py2.7/lib/python2.7/site-packages/gempython/scripts/anaUltraScurve.py", line 191, in <module>
tuple_calInfo = parseCalFile(options.calFile)
File "/home/dorney/venv_test/cc7/py2.7/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 419, in parseCalFile
import root_numpy as rp #note need root_numpy-4.7.2 (may need to run 'pip install root_numpy --upgrade')
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
File "/home/dorney/venv_test/cc7/py2.7/lib/python2.7/site-packages/root_numpy/__init__.py", line 51, in <module>
from ._tree import (
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
File "/home/dorney/venv_test/cc7/py2.7/lib/python2.7/site-packages/root_numpy/_tree.py", line 6, in <module>
from . import _librootnumpy
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
ImportError: libCore.so.5.34: cannot open shared object file: No such file or directory
See last line:
ImportError: libCore.so.5.34: cannot open shared object file: No such file or directory
Is this a potential issue with setupMachine.sh
maybe not taking the correct package version of root-numpy
?
Not sure, but let's decouple these issues:
pip install -I package
didn't reinstall dependencies, e.g., root_numpy
root_numpy
needs to be tracked down, i.e., why is it asking for some root5
bindings...After you tried to build the root_numpy
rpm
by hand it seems like a version bound to root5
is still being installed when setting up a venv
.
I tried with a fresh venv
and still obtained:
(py2.7) [dorney@gem904qc8daq]~/scratch0/CMS_GEM/CMS_GEM_DAQ% cd /data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/trim/z4.000000/2018.06.01.04.39
(py2.7) [dorney@gem904qc8daq]/data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/trim/z4.000000/2018.06.01.04.39% anaUltraScurve.py -i SCurveData_trimdac0.root -c -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Analyzing: 'SCurveData_trimdac0.root'
mkdir: cannot create directory ‘SCurveData_trimdac0’: File exists
Traceback (most recent call last):
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/lib/python2.7/site-packages/gempython/scripts/anaUltraScurve.py", line 189, in <module>
tuple_calInfo = parseCalFile(options.calFile)
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 419, in parseCalFile
import root_numpy as rp #note need root_numpy-4.7.2 (may need to run 'pip install root_numpy --upgrade')
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/lib/python2.7/site-packages/root_numpy/__init__.py", line 51, in <module>
from ._tree import (
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/lib/python2.7/site-packages/root_numpy/_tree.py", line 6, in <module>
from . import _librootnumpy
File "/usr/lib64/python2.7/site-packages/ROOT.py", line 318, in _importhook
return _orig_ihook( name, *args, **kwds )
ImportError: libCore.so.5.34: cannot open shared object file: No such file or directory
Below is the venv
setup.
venv
setup(py2.7) [dorney@gem904qc8daq]~/scratch0/CMS_GEM/CMS_GEM_DAQ% source setup_gemdaq.sh -c 0.3.1 -v 2.0.0 -V 3 -p $PWD/venv
Already using interpreter /usr/bin/python
New python executable in /afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7/bin/python
Installing setuptools, pip, wheel...done.
Requirement already up-to-date: importlib in /afs/cern.ch/user/d/dorney/.local/lib/python2.7/site-packages (1.0.4)
Requirement already up-to-date: setuptools in ./venv/cc7/py2.7/lib/python2.7/site-packages (39.2.0)
Requirement already up-to-date: pip in ./venv/cc7/py2.7/lib/python2.7/site-packages (10.0.1)
xstatic-jquery-ui 1.12.0.1 requires XStatic-jQuery, which is not installed.
sphinxcontrib-spelling 2.3.0 requires PyEnchant>=1.6.5, which is not installed.
pyramid 1.5.6 requires WebOb>=1.3.1, which is not installed.
sphinx-rtd-theme 0.1.8 has requirement sphinx>=1.3, but you'll have sphinx 1.1.3 which is incompatible.
rtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you'll have pyudev 0.15 which is incompatible.
ipapython 4.5.4 has requirement dnspython>=1.15, but you'll have dnspython 1.12.0 which is incompatible.
cp /afs/cern.ch/user/s/sturdy/public/cmsgemos_gempython-0.3.1.tar.gz .
python -m pip install cmsgemos_gempython-0.3.1.tar.gz --no-deps
Processing ./cmsgemos_gempython-0.3.1.tar.gz
Building wheels for collected packages: cmsgemos-gempython
Running setup.py bdist_wheel for cmsgemos-gempython ... done
Stored in directory: /afs/cern.ch/user/d/dorney/.cache/pip/wheels/33/e4/f0/e0ad3108a73ad75c2385987f6d2e37cc72fe034eb77451ebcb
Successfully built cmsgemos-gempython
Installing collected packages: cmsgemos-gempython
Found existing installation: cmsgemos-gempython 0.3.1
Not uninstalling cmsgemos-gempython at /usr/lib/python2.7/site-packages, outside environment /afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7
Can't uninstall 'cmsgemos-gempython'. No files were found to uninstall.
Successfully installed cmsgemos-gempython-0.3.1
wget https://github.com/cms-gem-daq-project/vfatqc-python-scripts/releases/download/v2.0.0-dev3/gempython_vfatqc-2.0.0.tar.gz
--2018-06-06 21:52:22-- https://github.com/cms-gem-daq-project/vfatqc-python-scripts/releases/download/v2.0.0-dev3/gempython_vfatqc-2.0.0.tar.gz
Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112
Connecting to github.com (github.com)|192.30.253.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/58535542/c83e9b00-5393-11e8-8c23-c1df69e3a595?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180606T195223Z&X-Amz-Expires=300&X-Amz-Signature=7e8a3b875f200935f054dd5481d39005b9a237b04e433618dcf4c2475575e781&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dgempython_vfatqc-2.0.0.tar.gz&response-content-type=application%2Foctet-stream [following]
--2018-06-06 21:52:23-- https://github-production-release-asset-2e65be.s3.amazonaws.com/58535542/c83e9b00-5393-11e8-8c23-c1df69e3a595?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180606%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180606T195223Z&X-Amz-Expires=300&X-Amz-Signature=7e8a3b875f200935f054dd5481d39005b9a237b04e433618dcf4c2475575e781&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dgempython_vfatqc-2.0.0.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.228.104
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.228.104|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21077 (21K) [application/octet-stream]
Saving to: ‘gempython_vfatqc-2.0.0.tar.gz’
100%[====================================================================================================================================================================================================>] 21,077 --.-K/s in 0.1s
2018-06-06 21:52:24 (196 KB/s) - ‘gempython_vfatqc-2.0.0.tar.gz’ saved [21077/21077]
python -m pip install gempython_vfatqc-2.0.0.tar.gz
Processing ./gempython_vfatqc-2.0.0.tar.gz
Requirement already satisfied: root_numpy in /usr/lib64/python2.7/site-packages (from gempython-vfatqc==2.0.0) (4.7.3)
Requirement already satisfied: numpy<1.15,>1.7 in /usr/lib64/python2.7/site-packages (from gempython-vfatqc==2.0.0) (1.7.1)
Building wheels for collected packages: gempython-vfatqc
Running setup.py bdist_wheel for gempython-vfatqc ... done
Stored in directory: /afs/cern.ch/user/d/dorney/.cache/pip/wheels/44/ce/e5/a4216b1bcd56625ab745c31a36a9fd96b28d95d770f518a2c6
Successfully built gempython-vfatqc
xstatic-jquery-ui 1.12.0.1 requires XStatic-jQuery, which is not installed.
sphinxcontrib-spelling 2.3.0 requires PyEnchant>=1.6.5, which is not installed.
pyramid 1.5.6 requires WebOb>=1.3.1, which is not installed.
sphinx-rtd-theme 0.1.8 has requirement sphinx>=1.3, but you'll have sphinx 1.1.3 which is incompatible.
rtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you'll have pyudev 0.15 which is incompatible.
ipapython 4.5.4 has requirement dnspython>=1.15, but you'll have dnspython 1.12.0 which is incompatible.
Installing collected packages: gempython-vfatqc
Found existing installation: gempython-vfatqc 2.0.0
Not uninstalling gempython-vfatqc at /usr/lib/python2.7/site-packages, outside environment /afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/cc7/py2.7
Can't uninstall 'gempython-vfatqc'. No files were found to uninstall.
Successfully installed gempython-vfatqc-2.0.0
Then installing gem-plotting-tools
from an rpm I built myself:
(py2.7) [dorney@gem904qc8daq]~/scratch0/CMS_GEM/CMS_GEM_DAQ% python -m pip install -I gem-plotting-tools/rpm/gempython_gemplotting-1.0.0.tar.gz
Processing ./gem-plotting-tools/rpm/gempython_gemplotting-1.0.0.tar.gz
Collecting root_numpy (from gempython-gemplotting==1.0.0)
Collecting numpy<1.15,>1.7 (from gempython-gemplotting==1.0.0)
Using cached https://files.pythonhosted.org/packages/05/7d/df63c9d42eb4485e79b61a054ba5db7c8efb830d29504d4fa849cf80eb01/numpy-1.14.4-cp27-cp27mu-manylinux1_x86_64.whl
Building wheels for collected packages: gempython-gemplotting
Running setup.py bdist_wheel for gempython-gemplotting ... done
Stored in directory: /afs/cern.ch/user/d/dorney/.cache/pip/wheels/13/ae/43/56ae2398e13083ccc7fe4823acca0b7bd6abe30924a94ae2d1
Successfully built gempython-gemplotting
xstatic-jquery-ui 1.12.0.1 requires XStatic-jQuery, which is not installed.
sphinxcontrib-spelling 2.3.0 requires PyEnchant>=1.6.5, which is not installed.
pyramid 1.5.6 requires WebOb>=1.3.1, which is not installed.
sphinx-rtd-theme 0.1.8 has requirement sphinx>=1.3, but you'll have sphinx 1.1.3 which is incompatible.
rtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you'll have pyudev 0.15 which is incompatible.
ipapython 4.5.4 has requirement dnspython>=1.15, but you'll have dnspython 1.12.0 which is incompatible.
Installing collected packages: root-numpy, numpy, gempython-gemplotting
Successfully installed gempython-gemplotting-1.0.0 numpy-1.14.4 root-numpy-4.7.3
Based on discussion in ops meeting I have tried:
[gemuser@gem904qc8daq tmp]$ python
Python 2.7.5 (default, Apr 11 2018, 07:36:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gempython.gemplotting.utils.anautilities import parseCalFile
>>> parseCalFile("/data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt")
TTree::SetCacheSizeAux:0: RuntimeWarning: A TTreeCache could not be created because the TTree has no file
(array([-0.2692 , -0.238106 , -0.25319999, -0.27678299, -0.28 ,
1. , -0.29872099, -0.246732 , -0.244578 , -0.24450199,
-0.27260301, -0.25 , 1. , -0.25592199, -0.26756299,
-0.25104499, -0.242985 , -0.24558499, -0.27000001, -0.27000001,
-0.274398 , -0.26885599, -0.25 , -0.238913 ]), array([ 66.01625061, 58.06386948, 62.37217331, 68.10211945,
69.41000366, 0. , 73.83969116, 60.39213943,
60.1219101 , 60.58520889, 66.92677307, 61.65999985,
0. , 62.92581558, 66.06646729, 61.58579636,
59.49411774, 60.52045441, 66.87999725, 66.62999725,
67.62643433, 66.5633316 , 61.54000092, 58.32714844]))
So indeed the system version of root-numpy
has no issue. Trying on a dummy venv
:
% source $BUILD_HOME/setup_gemdaq.sh -c 0.3.1 -g 1.0.0 -G 5 -p $PWD/venv_test 2>&1 | tee venvTestLog.log
% source $BUILD_HOME/setup_gemdaq.sh -p $PWD/venv_test
source /home/dorney/venv_test/cc7/py2.7/bin/activate
(py2.7) % python
Python 2.7.5 (default, Apr 11 2018, 07:36:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gempython.gemplotting.utils.anautilities import parseCalFile
>>> parseCalFile("/data/bigdisk/GEM-Data-Taking/GE11_QC8/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt")
TTree::SetCacheSizeAux:0: RuntimeWarning: A TTreeCache could not be created because the TTree has no file
(array([-0.2692 , -0.238106 , -0.25319999, -0.27678299, -0.28 ,
1. , -0.29872099, -0.246732 , -0.244578 , -0.24450199,
-0.27260301, -0.25 , 1. , -0.25592199, -0.26756299,
-0.25104499, -0.242985 , -0.24558499, -0.27000001, -0.27000001,
-0.274398 , -0.26885599, -0.25 , -0.238913 ]), array([ 66.01625061, 58.06386948, 62.37217331, 68.10211945,
69.41000366, 0. , 73.83969116, 60.39213943,
60.1219101 , 60.58520889, 66.92677307, 61.65999985,
0. , 62.92581558, 66.06646729, 61.58579636,
59.49411774, 60.52045441, 66.87999725, 66.62999725,
67.62643433, 66.5633316 , 61.54000092, 58.32714844]))
So it seems to work now.
Looks like my old venv
somehow kept some knowledge of the old root-numpy
version in which root 5 bindings were used. Probably a carry over.
Closing this issue.
Brief summary of issue
Parsing of a calibration file causes an exception. This was not an issue pre-packaging so I suspect perhaps there is still an issue with how
root_numpy
is installed on the machine.Types of issue
Expected Behavior
Parsing a calibration file should not cause an exception.
Current Behavior
Doesn't throw an import error which is strange. The tree2array function was definitely in 4.7.2 my
venv
has4.7.3
installed:This does not appear to be a problem with my
venv
since the user account on theqc8
machine gives the same problem:Steps to Reproduce (for bugs)
anaUltraScurve.py -i SomeSCurveFile.root -f --calFile=$DATA_PATH/GE11-X-S-CERN-0003/scurve/calFile_GE11-X-S-CERN-0003.txt
Possible Solution (for bugs)
The internet seems to think this is due to multiple top-level imports as described here. However I could not find an instance in which
root_numpy
was imported at top level inanaUltraScurve.py
or any of it's imports.Context (for feature requests)
Cannot run calibrated analysis.
Your Environment
1.0.0
/bin/zsh