missing shortChannelMap.txt error

AndrewLevin commented 6 years ago

Brief summary of issue

ana_scans.py script terminates normally but reports a python error.

Types of issue

Bug report

Expected Behavior

I have never successfully run the script before.

Current Behavior

[gemuser@gem904qc8daq gemdaq]$ ana_scans.py --scandate=current --ztrim=4 --anaType=scurve  2>&1 | tee -a ana_scurve_log.txt   
Running jobs in parallel mode (using Pool(12))
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Analysis Requested: scurve
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
Error: command exited with non-zero code 1
[1, 1, 1, 1]
Normal termination
Results: [1, 1, 1, 1]
Caught non-Python Exception <type 'exceptions.SystemExit'>
$ cat /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve/current/anaLog.log 
Analyzing: '/data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-VII-S-CERN-0001/scurve//current/SCurveData.root'
Exception: [Errno 2] No such file or directory: '/usr/lib/python2.7/site-packages/gempython/gemplotting/mapping//shortChannelMap.txt'
Failed to open: '/usr/lib/python2.7/site-packages/gempython/gemplotting/mapping//shortChannelMap.txt'
Traceback (most recent call last):
  File "/opt/cmsgemos/bin/anaUltraScurve.py", line 277, in <module>
    dict_vfatChanLUT = getMapping(MAPPING_PATH+'/shortChannelMap.txt')
  File "/usr/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 142, in getMapping
    mapFile.close()
UnboundLocalError: local variable 'mapFile' referenced before assignment

Steps to Reproduce (for bugs)

ssh to gemuser@gem904qc8daq from lxplus
execute "ana_scans.py --scandate=current --ztrim=4 --anaType=scurve 2>&1 | tee -a ana_scurve_log.txt"

Possible Solution (for bugs)

I have no idea.

Context (for feature requests)

This issue is preventing me from analyzing s-curve calibration data.

Your Environment

Version used: Not sure
Shell used: bash

mexanick commented 6 years ago

I see that the maps itself aren't included in the package distribution, but have to be built. @bdorney @jsturdy please either add map creation in the packaging routine or update the documentation reflecting the fact that user must create the maps before starting the analysis

bdorney commented 6 years ago

% ll /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/                    
total 68K
-rw-r--r--. 1 root root  907 May 18 10:38 PanChannelMaps.py
-rw-r--r--. 1 root root    0 May 18 10:38 __init__.py
-rw-r--r--. 1 root root 2.2K May 18 10:38 channelMaps.py
-rw-r--r--. 1 root root  696 May 18 10:38 buildMapFiles.py
-rw-r--r--. 1 root root  320 May 18 10:38 amcInfo.py
-rw-r--r--. 2 root root  978 May 18 10:38 PanChannelMaps.pyo
-rw-r--r--. 2 root root  978 May 18 10:38 PanChannelMaps.pyc
-rw-r--r--. 2 root root  161 May 18 10:38 __init__.pyo
-rw-r--r--. 2 root root  161 May 18 10:38 __init__.pyc
-rw-r--r--. 2 root root 2.4K May 18 10:38 channelMaps.pyo
-rw-r--r--. 2 root root 2.4K May 18 10:38 channelMaps.pyc
-rw-r--r--. 2 root root 1.5K May 18 10:38 chamberInfo.pyo
-rw-r--r--. 2 root root 1.5K May 18 10:38 chamberInfo.pyc
-rw-r--r--. 2 root root  946 May 18 10:38 buildMapFiles.pyo
-rw-r--r--. 2 root root  946 May 18 10:38 buildMapFiles.pyc
-rw-r--r--. 2 root root  568 May 18 10:38 amcInfo.pyo
-rw-r--r--. 2 root root  568 May 18 10:38 amcInfo.pyc
lrwxrwxrwx. 1 root root   62 May 18 10:40 chamberInfo.py -> /home/gemuser/gemdaq/gem-plotting-tools/mapping/chamberInfo.py

Seems the issue is that the mapping files are never built. The original assumption was that the old setup/paths.sh would call buildMapFiles.py:

https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/dd3c00f47844ac620279ac57e6dd1339d188583b/setup/paths.sh#L45-L50

The rpm installation procedure should do this or they should be included with the repo. In the short term perhaps @jsturdy could make a symlink for them here:

/home/gemuser/gemdaq/gem-plotting-tools/mapping/shortChannelMap.txt
/home/gemuser/gemdaq/gem-plotting-tools/mapping/longChannelMap.txt

bdorney commented 6 years ago

See updated comment above.

AndrewLevin commented 6 years ago

The relevant files have mysteriously appeared now:

[gemuser@gem904qc8daq gemdaq]$ ll /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/ | grep ChannelMap.txt lrwxrwxrwx. 1 root root 67 May 25 16:56 shortChannelMap.txt -> /home/gemuser/gemdaq/gem-plotting-tools/mapping/shortChannelMap.txt lrwxrwxrwx. 1 root root 66 May 25 16:56 longChannelMap.txt -> /home/gemuser/gemdaq/gem-plotting-tools/mapping/longChannelMap.txt

We don't understand why they were not there before and the root cause of the problem, so we are leaving the issue open.

But this is not blocking us from doing the analysis of the s-curve scans anymore.

bdorney commented 6 years ago

The relevant files have mysteriously appeared now: ... ... We don't understand why they were not there before and the root cause of the problem, so we are leaving the issue open.

They were placed as a symlink by the system administer. The root of the problem is understood as described by myself and @mexanick above: the mapping files were not built by buildMapFiles.py at the time the rpm was installed.

Issue to remain open.

jsturdy commented 6 years ago

Hotfix linked will generate the current long/short map files on an RPM install. Doing a pip install, the files will still need to be generated, and this can be kept open if there is a nice mechanism inside setuptools to achieve the same thing as a %post RPM directive. This should be revisited in the future as having "databases" of text files is not ideal long-term

cms-gem-daq-project / gem-plotting-tools