Importable DAC Scan Analysis Routine & DB Tools

bdorney commented 5 years ago

Description

Basic idea behind this PR was to provide basic DB functionality for calibration routines in lab and to provide importable analysis routine for DAC scans.

Detailed summary of changes:

anaDACScan.py now imports the dac analysis routine from utils/anautilities.py
Function fitScanData(...) can now take either a calibration file to be parsed or a pre-parsed tuple. It will check for the calibration file first (until reed-muller is implemented...),
Provided DB utils for returning views relevant for ASIC calibration.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context

Connectivity testing provides a place for DAC scans to be done in situ. Since a follow-up scurve is expected to check the number of dead channels are within tolerance it is also expected that the DAC scans be analyzed in situ and then results applied. This means an importable analysis routine and DB reading is required.

How Has This Been Tested?

Changes to DAC Analysis

21 Feb 2019 23:49:14.724 [7f59d6e8a740] INFO - wrappers::runCommand <> - Executing command: mkdir -p /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41
21 Feb 2019 23:49:15.963 [7f59d6e8a740] INFO - wrappers::runCommand <> - Executing command: chmod g+rw /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41
21 Feb 2019 23:49:15.997 [7f59d6e8a740] INFO - wrappers::runCommand <> - Executing command: unlink /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/current
21 Feb 2019 23:49:16.029 [7f59d6e8a740] INFO - wrappers::runCommand <> - Executing command: ln -s /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41 /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/current
TTree::SetCacheSizeAux:0: RuntimeWarning: A TTreeCache could not be created because the TTree has no file
Initializing TObjects
Looping over stored events in dacScanTree
fitting DAC vs. ADC distributions
Determining nominal values for bias voltage and/or current settings
Warning: when fitting VFAT9 of chamber GE11-X-L-CERN-0002 (OH0) DAC CFG_HYST the fitted value, -1, is outside range the register can hold: [0,63]. It will be replaced by 0.
Warning: when fitting VFAT10 of chamber GE11-X-L-CERN-0002 (OH0) DAC CFG_HYST the fitted value, -2, is outside range the register can hold: [0,63]. It will be replaced by 0.
Writing output data
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created
Info in <TCanvas::Print>: png file /data/bigdisk/GEM-Data-Taking/GE11_QC8//GE11-X-L-CERN-0002/dacScans/2019.02.21.23.41/SummaryGE11-X-L-CERN-0002_DACScan_2019.02.21.23.41.png has been created

Works as expected.

Testing DB Interaction Tools

Using test script I queried the VFAT ChipID's and then tried to look in the DB to retrieve their calibration values:

% python dbTester.py --shelf=1 -s2 -g3
Open pickled address table if available  /opt/cmsgemos/etc/maps/amc_address_table_top.pickle...
Initializing AMC gem-shelf01-amc02
0 4713
1 4715
2 5905
3 3715
4 3750
5 4730
6 268702319
7 3540
8 2943
9 6079
10 68411
11 4826
12 2696
13 2876
14 136969
15 3712
16 4868
17 4394
18 4171
19 4861
20 4968
21 4395
22 5948
23 4176
Length of returned view does not match length of input vfat List
VFATs not found: ['0x127a', '0x1004126f', '0x10b3b', '0x21709', '0x1368']
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 19 entries, 0 to 18
Data columns (total 9 columns):
vfat3_ser_num    19 non-null object
vfat3_barcode    19 non-null object
iref             19 non-null int64
adc0m            19 non-null float64
adc1m            19 non-null float64
adc0b            19 non-null float64
adc1b            19 non-null float64
cal_dacm         19 non-null float64
cal_dacb         19 non-null float64
dtypes: float64(6), int64(1), object(2)
memory usage: 1.4+ KB
goodbye

So we can read the data in the DB. Reason for some VFATs not being found is due to the chipID bit flip problem; successive reads show different number of VFATs not found and a different size of the output dataframe. You can see the bit flip clearly in the printed list in decimal,

Screenshots (if appropriate):

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

jsturdy commented 5 years ago

cc7 defaults installed on gem904daq01 and gem904qc8daq for:

python-sqlalchemy
python-pandas
cx_Oracle

bdorney commented 5 years ago

ready for review @jsturdy @mexanick

bdorney commented 5 years ago

@jsturdy removed hard coded names and switched to env variables, typos fixed, and then all retested.

I think we are good now.

cms-gem-daq-project / gem-plotting-tools