Feature Updated ana_scans.py and made most analysis routines importable

[bdorney]

Description

Addresses issues:

• https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/188 (Partially, for scurves)
• https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/202
• https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/216

This has been a significant update to the package. Summary of changes is presented below:

• In most cases optparse.OptionParser has been abandoned in favor of argparse.ArgumentParser; this follows from past discussions with @jsturdy.
  • Additionally I've tried to strip all command line option/arguments that had no effect from a script (e.g. caused by blanket lazy imports of options menus).
• The scurve analysis will now also produce some summary distributions for effective pedestal.
• Provided dedicated function for determining scandate from filename: getScandateFromFilename() in utils/anautilities.py.
• Provided dedicated function for determing GEBType from filename: getGEBTypeFromFilename() in utils/anautilities.py
• Provided dedicated function for determing chamber name from filename: getChamberNameFromFilename() in utils/anautilities.py
• The following routines have made the bulk of the algorithm moved to a dedicated file under utils/ so that the analysis routine is importable (with in mind of enabling more automation in QC7):
  • anaUltraLatency() defined in utils/latAlgos.py
  • fill2DScurveSummaryPlots(), anaUltraScurve(), plotAllSCurvesOnCanvas() to utils/scurveAlgos.py
  • anaUltraThreshold() and calibrateThrDAC() to utils/threshAlgos.py
• The sbitRateAnalysis routine has been moved from utils/anautilities.py to utils/threshAlgos.py
• Many generic exception handling statements (e.g. except Exception as e) have been changed to specific exception types for more appropriate error handling.
• Unused file mapping/amcInfo.py with hard coded uTCA shelf layout has been deleted.
• Dictionaries ana_config and tree_names have had newer analysis routines added to them and now use templated strings where appropriate.
• Updated ana_scans.py for new tools with v3 electronics in mind; additionally all processes launched use importable analysis routines so additional process are owned by the parent process. Killing the parent process will kill all child processes (this was not happening in previous implementation and child processes needed to be killed manually).
  • Added a utility function getNumCores2Use() in utils/anautilities.py for determining the CPU load ana_scans.py should take when running parallel processes.
  • Added init_worker() function to allow child processes to have KeyboardInterrupt's handled by the parent process.
  • For analysis routines of VFAT tracking data launched with python tools the stdout and stderr will be redirected to a dedicated log file if the calling process is not the parent process.
• Added *.rst files to help expand existing documentation.
• It is now possible to analyze only a subset of the VFATs in an scurve file; default behavior is to analyze all VFATs.
• If the --calFile argument is not provided to anaUltraScurve.py at runtime the script will use the data in the vfatID TBranch of the input TTree to query the DB to retrieve the calibration coefficients. This partially addresses: https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/188.
• Based on improvements to GND schema of detector+QC8 and analyzing noise data of detectors default scurve cuts have been changed too:
  • Dead channel low and high cut have been defaulted to 0.0 fC until such time as we have a reliable method for distinguishing again,
  • High noise cut decreased to 1.5 fC from 3.0 fC
  • Effective Pedestal cut decreased to 2% from 5%.
  • The cut for the MAD algorithm has been increased from 3.5 to 10.0 to address overperforming issues of the algorithm (this in principle should at most now remove 1% of total outliers); before it was over-cutting.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [x] Breaking change (fix or feature that would cause existing functionality to change)

The usage of the following files has changed:

• For anaSBitMonitor.py and anaUltraLatency.py the infilename argument is now a positional argument.
• For anaUltraScurve.py and anaUltraThreshold.py the infilename and GEBtype arguments are now positional arguments.

Additionally significant changes to ana_scans.py have been made which are described in depth below.

Motivation and Context

QC8 shifters where either:

• launching analysis of multiple data in series with a giant copy/pasted script, or
• Opening multiple terminals to launch the same command on a series of different input files manually.

For efficient usage of time and computing resources I've provided an update to ana_scans.py to address these concerns.

Also, with the goal to automate further QC7 and reduce mistakes we need the ability to import analysis of several scans (e.g. scurve, tracking data threshold, and sbit threshold) into the QC7 routine. These changes allow for that.

Finally, since run_scans.py provides (with the exception of calibration of CFG_THR_ARM_DAC) a single entry point to the calibration measurement program we should have a similar tool for analysis data for QOL improvements.

Updates to ana_scans.py

The script has been re-designed to make use of sub command parsers. The script now features multiple subcommands and where possible these sub commands use the same keyword as the run_scan.py equivalent to provide easy recognition to the shifter.

The following analysis are not implemented (no change from past behavior) for the equivalent call of the following tools:

• anaSBITMonitor.py,
• anaSBitReadout.py,
• anaXDAQLatency.py,

However these tools can't really generate data that can be analyzed in parallel (or a tool already exists to do so, e.g. LDQM for anaXDAQLatency.py).

Master help menu

% ana_scans.py -h
usage: ana_scans.py [-h]
                    {armDacCal,dacScanV3,lat,sbitThresh,scurve,thrDac,trim}
                    ...

Arguments to supply to ana_scans.py

positional arguments:
  {armDacCal,dacScanV3,lat,sbitThresh,scurve,thrDac,trim}
                        Available subcommands and their descriptions. To view
                        the sub menu call ana_scans.py COMMAND -h
                        e.g. 'ana_scans.py dacScanV3 -h'
    armDacCal           Uses the anaUltraScurve.py and calibrateThrDac.py
                        tools to analyze calibration data of CFG_THR_ARM_DAC.
                        This will first check if all scurves in the input
                        listOfScandates files specified by the scandate or
                        inputfilename option have been analyzed. If not it
                        will analyze them all in parallel. Then it will
                        perform all calibration analyses in parallel
    dacScanV3           Uses the anaDACScan.py tool to analyze VFAT3 DAC scan
                        data
    lat                 Uses the anaUltraLatency.py tool to analyze latency
                        scans taken with either ultraLatency or 'run_scans.py
                        lat'
    sbitThresh          Analyzes scurve data taken with either
                        sbitThreshScanParallel.py or 'run_scans sbitThresh'
    scurve              Analyzes scurve data taken with either ultraScurve.py
                        or 'run_scans.py scurve'
    thrDac              Analyzes tracking data threshold scans taken with
                        either ultraThreshold.py or 'run_scans.py thrDac'
    trim                Analyzes scurves taken with trimChamberV3.py or
                        `run_scans.py trim'

optional arguments:
  -h, --help            show this help message and exit

The armDacCal command

% ana_scans.py armDacCal -h
usage: ana_scans.py armDacCal [-h] [-d]
                              (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                              (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                              (--light | --medium | --heavy)
                              [--maxEffPedPercent MAXEFFPEDPERCENT]
                              [--highNoiseCut HIGHNOISECUT]
                              [--deadChanCutLow DEADCHANCUTLOW]
                              [--deadChanCutHigh DEADCHANCUTHIGH] [-c | -p]
                              [-z ZSCORE] [--delimiter DELIMITER]
                              [--doNotSavePlots] [-n]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  --light               Analysis uses only 25% of available cores
  --medium              Analysis uses only 50% of available cores
  --heavy               Analysis uses only 75% of available cores
  -c, --channels        Make plots vs channels instead of strips
  -p, --panasonic       Make plots vs Panasonic pins instead of strips
  -z ZSCORE, --zscore ZSCORE
                        Z-Score for Outlier Identification in MAD Algo, used
                        to set HotChannel bit
  --delimiter DELIMITER
                        Character used to delimit the listOfScandates files
                        that will be analyzed with the scandate or
                        inputfilename option
  --doNotSavePlots      Add this argument if you do *not* want to save all
                        *.png files
  -n, --noLeg           Add this option if you do not want output plots to
                        have a legend drawn on them

Options for channel mask decisions:
  Parameters which specify how Dead, Noisy, and High Pedestal Channels are
  charaterized

  --maxEffPedPercent MAXEFFPEDPERCENT
                        Percentage, threshold for setting the HighEffPed mask
                        reason, if channel (effPed > maxEffPedPercent * nevts)
                        then HighEffPed is set
  --highNoiseCut HIGHNOISECUT
                        Threshold in fC for setting the HighNoise maskReason,
                        if channel (scurve_sigma > highNoiseCut) then
                        HighNoise is set
  --deadChanCutLow DEADCHANCUTLOW
                        If channel (deadChanCutLow < scurve_sigma <
                        deadChanCutHigh) then DeadChannel is set
  --deadChanCutHigh DEADCHANCUTHIGH
                        If channel (deadChanCutHigh < scurve_sigma <
                        deadChanCutHigh) then DeadChannel is set

The dacScanV3 command

% ana_scans.py dacScanV3 -h
usage: ana_scans.py dacScanV3 [-h] [-d]
                              (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                              (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                              [--calFileList CALFILELIST] [-o OUTFILENAME]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  --calFileList CALFILELIST
                        Optional. File specifying which calFile to use for
                        each OH. Format of each line: <shelf> <slot> <link>
                        /path/to/my/cal/file.txt If this is not provided the
                        algorithm will search
                        '$DATA_PATH/DetectorName/calFile_ADC0_DetectorName.txt
  -o OUTFILENAME, --outfilename OUTFILENAME
                        Filename to which output information is written

The lat Command

% ana_scans.py lat -h
usage: ana_scans.py lat [-h] [-d]
                        (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                        (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                        (--light | --medium | --heavy) [-f]
                        [--latSigRange LATSIGRANGE]
                        [--latSigMaskRange LATSIGMASKRANGE]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  --light               Analysis uses only 25% of available cores
  --medium              Analysis uses only 50% of available cores
  --heavy               Analysis uses only 75% of available cores
  -f, --fit             Fit the latency distributions
  --latSigRange LATSIGRANGE
                        Comma separated pair of values defining expected
                        signal range, e.g. lat #epsilon [41,43] is signal
  --latSigMaskRange LATSIGMASKRANGE
                        Comma separated pair of values defining the region to
                        be masked when trying to fit the noise, e.g. lat
                        #notepsilon [40,44] is noise (lat < 40 || lat > 44)

The sbitThresh Command

% ana_scans.py sbitThresh -h
usage: ana_scans.py sbitThresh [-h] [-d]
                               (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                               (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                               [-m MAXNOISERATE] [--perchannel]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  -m MAXNOISERATE, --maxNoiseRate MAXNOISERATE
                        Max Noise Rate allowed in Hz
  --perchannel          If provided files to be analyzed will be treated as if
                        they are a perchannel scan; otherwise they will be
                        treated as if they where acquired taking all channels
                        for a given VFAT in OR

The scurve Command

% ana_scans.py scurve -h
usage: ana_scans.py scurve [-h] [-d]
                           (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                           (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                           (--light | --medium | --heavy)
                           [--maxEffPedPercent MAXEFFPEDPERCENT]
                           [--highNoiseCut HIGHNOISECUT]
                           [--deadChanCutLow DEADCHANCUTLOW]
                           [--deadChanCutHigh DEADCHANCUTHIGH] [-c | -p]
                           [-z ZSCORE] [--doNotFit]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  --light               Analysis uses only 25% of available cores
  --medium              Analysis uses only 50% of available cores
  --heavy               Analysis uses only 75% of available cores
  -c, --channels        Make plots vs channels instead of strips
  -p, --panasonic       Make plots vs Panasonic pins instead of strips
  -z ZSCORE, --zscore ZSCORE
                        Z-Score for Outlier Identification in MAD Algo, used
                        to set HotChannel bit
  --doNotFit            Do not attempt to fit the scurves

Options for channel mask decisions:
  Parameters which specify how Dead, Noisy, and High Pedestal Channels are
  charaterized

  --maxEffPedPercent MAXEFFPEDPERCENT
                        Percentage, threshold for setting the HighEffPed mask
                        reason, if channel (effPed > maxEffPedPercent * nevts)
                        then HighEffPed is set
  --highNoiseCut HIGHNOISECUT
                        Threshold in fC for setting the HighNoise maskReason,
                        if channel (scurve_sigma > highNoiseCut) then
                        HighNoise is set
  --deadChanCutLow DEADCHANCUTLOW
                        If channel (deadChanCutLow < scurve_sigma <
                        deadChanCutHigh) then DeadChannel is set
  --deadChanCutHigh DEADCHANCUTHIGH
                        If channel (deadChanCutHigh < scurve_sigma <
                        deadChanCutHigh) then DeadChannel is set

The thrDac Command

% ana_scans.py thrDac -h
usage: ana_scans.py thrDac [-h] [-d]
                           (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                           (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                           (--light | --medium | --heavy) [-c | -p]
                           [-z ZSCORE] [--pervfat]
                           [--scurveScandate SCURVESCANDATE]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  --light               Analysis uses only 25% of available cores
  --medium              Analysis uses only 50% of available cores
  --heavy               Analysis uses only 75% of available cores
  -c, --channels        Make plots vs channels instead of strips
  -p, --panasonic       Make plots vs Panasonic pins instead of strips
  -z ZSCORE, --zscore ZSCORE
                        Z-Score for Outlier Identification in MAD Algo, used
                        to set HotChannel bit
  --pervfat             Analysis for a per-VFAT scan (default is per-channel)
  --scurveScandate SCURVESCANDATE
                        Provide this option if you want the chConfig.txt
                        file(s) that is(are) produced to taken into account
                        the channel register data from a previously analyzed
                        set of scurves that share this scandate, in
                        YYYY.MM.DD.hh.mm forma.

The trim Command

% ana_scans.py trim -h
usage: ana_scans.py trim [-h] [-d]
                         (-s SCANDATE | -i INPUTFILENAME | -l LISTOFSCANDATESFILE)
                         (--chamberConfig | --cNameAndAddr CNAMEANDADDR | --mappingFile MAPPINGFILE)
                         (--light | --medium | --heavy)
                         [--maxEffPedPercent MAXEFFPEDPERCENT]
                         [--highNoiseCut HIGHNOISECUT]
                         [--deadChanCutLow DEADCHANCUTLOW]
                         [--deadChanCutHigh DEADCHANCUTHIGH] [-c | -p]
                         [-z ZSCORE] [--trimPoints TRIMPOINTS]

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Print additional debugging information
  -s SCANDATE, --scandate SCANDATE
                        scandate in YYYY.MM.DD.hh.mm format of input data.
                        Will find all files associated to this scandate for
                        the relevant command.
  -i INPUTFILENAME, --inputfilename INPUTFILENAME
                        Specify single filename to analyze
  -l LISTOFSCANDATESFILE, --listOfScandatesFile LISTOFSCANDATESFILE
                        Specify listOfScandates file. Note this is not
                        supported for commands armDacCal, dacScanV3 or
                        sbitThresh. The input file here is a tab delimited
                        file where the first line is the column header
                        'ChamberName Scandate' and subsequent lines are
                        ChamberName and scandate pairs, e.g.
                        'GE11-X-S-BARI-0013 YYYY.MM.DD.hh.mm'
  --chamberConfig       Use system specific chamber_config and GEBtype
                        dictionaries
  --cNameAndAddr CNAMEANDADDR
                        Comma separated string used to build chamber_config
                        and GEBtype dictionaries at runtime; format expected:
                        detector S/N, detector type, uTCA shelf, AMC slot, and
                        OH link, e.g. 'GE11-X-S-CERN-0001,short,1,4,7'.
  --mappingFile MAPPINGFILE
                        Comma separated file used to build chamber_config and
                        GEBtype dictionaries at runtime; each line gives
                        respectively detector S/N, detector type, uTCA shelf,
                        AMC slot, and OH link
  --light               Analysis uses only 25% of available cores
  --medium              Analysis uses only 50% of available cores
  --heavy               Analysis uses only 75% of available cores
  -c, --channels        Make plots vs channels instead of strips
  -p, --panasonic       Make plots vs Panasonic pins instead of strips
  -z ZSCORE, --zscore ZSCORE
                        Z-Score for Outlier Identification in MAD Algo, used
                        to set HotChannel bit
  --trimPoints TRIMPOINTS
                        Comma separated list of trim values that where used
                        when trimming, e.g. '-63,0,63'. Note a negative number
                        implies the trimPol register was set to 1 instead of
                        0. If not provided only the final 'Trimmed' scurve in
                        a trim run will be analyzed. This has no effect
                        if paired with the `--inputfilename` argument

Options for channel mask decisions:
  Parameters which specify how Dead, Noisy, and High Pedestal Channels are
  charaterized

  --maxEffPedPercent MAXEFFPEDPERCENT
                        Percentage, threshold for setting the HighEffPed mask
                        reason, if channel (effPed > maxEffPedPercent * nevts)
                        then HighEffPed is set
  --highNoiseCut HIGHNOISECUT
                        Threshold in fC for setting the HighNoise maskReason,
                        if channel (scurve_sigma > highNoiseCut) then
                        HighNoise is set
  --deadChanCutLow DEADCHANCUTLOW
                        If channel (deadChanCutLow < scurve_sigma <
                        deadChanCutHigh) then DeadChannel is set
  --deadChanCutHigh DEADCHANCUTHIGH
                        If channel (deadChanCutHigh < scurve_sigma <
                        deadChanCutHigh) then DeadChannel is set

How Has This Been Tested?

Extensive testing on 904 DAQ machines using old data.

Screenshots (if appropriate):

Checklist:

• [x] My code follows the code style of this project.
• [x] My change requires a change to the documentation.
• [x] I have updated the documentation accordingly.
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.

[jsturdy]

Feature Updated ana_scans.py #225 and Files changed: 30 really don't belong in the same sentence...

[bdorney]

rebased

[bdorney]

A few comments on some common blocks, nothing obviously major, but overall a bit difficult to separate the refactored parts from the new parts (refactor here means code that was moved, and OptParse -> ArgParse. This could be alleviated at review time by squashing (and reordering) commits related to specific changes, as code can be reviewed on a per-commit basis), will probably make a stronger recommendation in the future for this, since I'm also guilty of this.

I did not know it was possible to reorder commits; I have made many squashes throughout the development process. I think I squashed at least 20 commits here; perhaps I should have been more vigorous use of squash.

Second point, and probably more visible only due to the refactor, is the inconsistent usage of string formatting. While I don't expect all old formatting to be updated, I would expect a consistent implementation for any new code (with the expectation that at some point all old code will be cleaned up)

In all new cases I have tried to use exclusively str::format() method for string formatting/concatenation. I agree 100% that we should remove and reduce inconsistency. I think you are seeing a lot of formatting cases with the % operator because of the refactoring. If we want to open a new issue to address inconsistencies in string formatting.

[jsturdy]

@bdorney please bump when ready for re-review

[bdorney]

@bdorney please bump when ready for re-review

Yes I will do so; I have not implemented the proposed change here though.

[bdorney]

@bdorney please bump when ready for re-review

Ready for re-review @jsturdy and @mexanick

[bdorney]

rebased

[bdorney]

Assumption is that this is working Don't do a plain merge if this is approved, if merging, do a squash merge

I defaulted this repo to "squash and merge"