Feature Requests: Unit Tests

bdorney commented 7 years ago

Brief summary of issue

In the past week we have let PR's be merged without either proper testing or proper oversight (myself included) as demonstrated by:

https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/16 https://github.com/cms-gem-daq-project/gem-plotting-tools/issues/24

Which were either not properly tested originally, or not re-tested after committing on top of a request. To prevent lost time, and most importantly p5 machine being updated with bad code we should finalize a procedure and develop unit tests which should cross-check things. I propose:

Each PR should be supported with an e-log link that shows tests were properly done.
New commits after changes are requested should include an e-log link where tests have been repeated and properly passed.

Types of issue

[ ] Bug report (report an issue with the code)
[X] Feature request (request for change which adds functionality)

Expected Behavior

I would propose the following:

For changes to ana_scans.py for each --anaType a re-analysis of old data should be launched
For changes to anaUltra*.py scripts see the first item,
For changes to anaXDAQ*.py scripts see the first item,
For changes to files in fitting/ or mapping/ sub-directories taking a run of vfatqc-python-scripts/ultraScurve.py and then analyzing the data immediately

For items 1-3 something like scripts/unitTests_Ana_Ultra.sh script like:

#!/bin/bash

# record branch name and commit hash
cd $GEM_PLOTTING_PROJECT
git status 2>&1 | tee -a unit_test_log.txt
git show 2>&1 | tee -a unit_test_log.txt

# perform tests
ana_scans.py --scandate=current --anaType=latency  2>&1 | tee -a unit_test_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=scurve  2>&1 | tee -a unit_test_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=threshold  2>&1 | tee -a unit_test_log.txt
sleep 2 
ana_scans.py --scandate=current --anaType=trim  2>&1 | tee -a unit_test_log.txt

# Done
echo "Unit Tests Completed, Upload:"
echo $PWD/unit_test_log.txt
echo "To the E-Log or Github PR along with the images in:"
echo $ELOG_PATH

With a similar script e.g. scripts/unitTests_Ana_XDAQ.sh

These scripts should be launched on a test stand for at least 1 link in chamber_config of mapping/chamberInfo.py

For item 4 a script like scripts/unitTests_Run_SCurve.sh:

#!/bin/bash

# record branch name and commit hash
cd $GEM_PLOTTING_PROJECT
git status 2>&1 | tee -a unit_test_log.txt
git show 2>&1 | tee -a unit_test_log.txt

# perform tests
run_scans.py --tool=ultraScurve.py --shelf=$1 -s$2 2>&1 | tee -a scurve_log.txt
sleep 2
ana_scans.py --scandate=current --anaType=scurve  2>&1 | tee -a unit_test_log.txt
sleep 2

# Done
echo "Unit Tests Completed, Upload:"
echo $PWD/unit_test_log.txt
echo "To the E-Log or Github PR along with the images in:"
echo $ELOG_PATH

With syntax:

./scripts/unitTests_Run_SCurve.sh <Shelf> <Slot>

Current Behavior

We lack an appropriate unit test procedure.

Context (for feature requests)

We've committed things without proper foresight and testing. This should at least make the developer go through a procedure to check for problems.

lmoureaux commented 7 years ago

I'm looking into setting up Travis CI for gem-plotting-tools. It does automatic testing in the cloud and can block merging of failing branches. Preliminary results can be found at https://travis-ci.org/lmoureaux/gem-plotting-tools/builds

bdorney commented 7 years ago

Travis CI appears to be an available service that can be integrated into the github repo by an administrator. Can you summarize what is necessary to set this up and what benefits it will give? Also should this replace, or work in conjunction with, landscape?

lmoureaux commented 7 years ago

Travis is free to use for open source projects. It is intended for continuous integration, i.e. it can build source code (not relevant here) and run unit tests. Setting it up can is done by adding a .travis.yml file to the repo to describe the required environment, and one or more testing scripts (usually in .travis/). In order to use Travis CI, the organization should open an account (involves accepting Terms of Service and so on). Travis can then be configured to test every commit. Github can block failing pull requests, similar to what landscape does. Travis can be combined with landscape.

bdorney commented 6 years ago

Since WIP and travis CI are now implemented I am closing this issue. It can be reopened if necessary.

cms-gem-daq-project / gem-plotting-tools