[RFC] Integrate gem-plotting-tools with Travis CI

[lmoureaux]

Travis CI is an online platform for continuous integration, free-to-use for open source projects. It is deeply integrated with Github and can block merge just like landscape can. This PR adds support for Travis CI for gem-plotting-tools.

Description

Travis is very generic and can be told to run arbitrary commands. The "build" fails if any of the commands fails, which is the ideal behavior for unit tests. Travis uses Ubuntu 14.04 as their default environment, and I don't change this.

This PR adds a .travis.yml file to enable Travis. This file and the scripts in .travis setup the following testing environment:

• A Linux machine based on Ubuntu 14.04 (Travis' default; some cosmetic changes were needed for the code to run there)
• ROOT packages from the ROOT website [edited: was Ubuntu's ROOT packages]
• Python 2.6 and python 2.7 (tests will be run in both environments)
• gem-plotting-tools' dependencies (as per requirements.txt)
• cmsgemos from the default branch [edited: was master]
• cmsgemos' dependencies (as per requirements.txt) [edited: new]

In addition, this PR adds a test of ana_scans.py --runType=scurve. The test will catch syntax errors and fatal exceptions in ana_scans.py and anaUltraScurve.py, as well as any file they import. The test will not catch other problems. Some changes to ana_scans.py were required in order to make it fail when the subcommands fail.

If this PR gets merged, an administrator can subscribe to Travis and enable it within minutes.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context

See #26 for context. Having Travis run every tool for every commit would catch code typos early.

How Has This Been Tested?

The changes were continuously tested on Travis itself (and only there). See here for an example of the test running without error, here for the output of Travis catching bug #24.

Screenshots

This is the current Travis result for my branch travis...

Checklist:

• [x] My code follows the code style of this project.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.

[lmoureaux]

This PR isn't ready yet, but I'd like to get comments on the architecture before more tests get added to it.

[bdorney]

Ubuntu's ROOT packages

Can you confirm this is:

ROOT 5.34/36 (heads/v5-34-00-patches@v5-34-34-76-g57bae4c, Apr 08 2016, 12:05:00 on linuxx8664gcc)

Python 2.6 and python 2.7 (tests will be run in both environments):

Can you confirm these are:

Python 2.6.6
Python 2.7.5

As these are the defaults on slc6 and cc7 operating systems.

cmsgemos from branch master

Please use generic-amc-release-v3 of cmsgemos instead, found here.

One question. How does Travis CI update? e.g. if there is an update to cmsgemos what does one do to ensure that this update is included?

Also I think we need to expand this unit testing:

In addition, this PR adds a test of ana_scans.py --runType=scurve. The test will catch syntax errors and fatal exceptions in ana_scans.py and anaUltraScurve.py, as well as any file they import. The test will not catch other problems. Some changes to ana_scans.py were required in order to make it fail when the subcommands fail.

Let's expand this to include:

• anaUltraThreshold.py
• anaUltraLatency.py
• anaXDAQLatency.py

Also can the input files that we use as the "test data" be stored in the same file structure that ana_scans.py expects. This could also help us test chamberInfo.py.

[lmoureaux]

Ubuntu's ROOT packages

Can you confirm this is:

ROOT 5.34/36 (heads/v5-34-00-patches@v5-34-34-76-g57bae4c, Apr 08 2016, 12:05:00 on linuxx8664gcc)

amd64 root-system-bin amd64 5.34.14-1build1

Same architecture, major and minor version. Patch number is different, though. I don't know of a simple way to install a specific ROOT version on Debian (and I don't want to compile it).

Can you confirm these are:

Python 2.6.6 Python 2.7.5

Python 2.6.13 and 2.7.13 (more recent patch versions, no behavior change expected). I can look into specifying patch numbers, but I'm not sure whether Travis still supports 2.6.6 (since even 2.6.13 is deprecated as it's not supported by the Python team anymore).

cmsgemos from branch master

Please use generic-amc-release-v3 of cmsgemos instead, found here.

Checked, it uses the default branch, which is generic-amc-release-v3 for now. Should I specify the branch explicitly?

One question. How does Travis CI update? e.g. if there is an update to cmsgemos what does one do to ensure that this update is included?

The latest cmsgemos if fetched from git every time the test is run.

[bdorney]

Same architecture, major and minor version. Patch number is different, though. I don't know of a simple way to install a specific ROOT version on Debian (and I don't want to compile it).

Compiled sources are available, for example for v5.34.36: https://root.cern.ch/content/release-53436

Checked, it uses the default branch, which is generic-amc-release-v3 for now. Should I specify the branch explicitly?

No let's rely on the correct setup of the cmsgemos repo to point git to this branch as default. This can only be changed by an admin of the cmsgemos repo which I think includes only @jsturdy presently.

The latest cmsgemos if fetched from git every time the test is run.

Yeah I saw after looking at the code.

[jsturdy]

Sorry to not jump on this earlier, but it would be better if you can set it up to work with the docker container I set up for cmsgemos CI (work in progress, and once it's sorted, the plan was to simply plug in the other packages)

• Here's the travis.yml I've been working with: https://github.com/jsturdy/cmsgemos/blob/jsturdy-travis-int-test/.travis.yml (I also have templates in vfatqc-python-scripts
• Here's the docker I currently test against: https://gitlab.cern.ch/sturdy/gemdaq_ci_worker, though I was having some issues getting things working the way I wanted from github and moved to other tasks with higher priority

Other things to probably keep in mind moving forward:

• Using the docker container (which can be updated on an as-needed basis) allows to maintain a testing environment that will be as close to operation as possible
  • I will update the container semi-regularly to make sure it mimics the machines we typically run software on
  • The container can even be set up to have a directory structure that hosts some dummy data to run the unit tests against
• Testing against the "latest" cmsgemos isn't the best practice, but can suffice for now. At a future point you can add the gempython package (to be) produced by cmsgemos to the list of requirements, and then specify a version
• You can also look into using tox for python unit testing, which integrates easily with travis, though I've not looked deeply into what it offers over simple unit testing run inside travis

[lmoureaux]

In today's update:

• Using ROOT binaries downloaded directly from the ROOT website (v5.34.36)
• Install cmsgemos' Python dependencies (none atm)
• Cleanup .gitignores as requested
• The test script now checks that all output files are present and non-empty
• Ensures that tests are run from $DATA_PATH
• Changes to ana_scans.py were moved to PR #29. The code is kept here because it's needed for this branch to work properly. I will revert just before merging master once PR #29 gets merged.

I considered using @jsturdy's Docker image, but I couldn't even run ls from inside the container. As I don't have the expertise to fix it (both on Docker and on how to setup cmsgemos), I went ahead with Ubuntu as the second-best solution. The tests themselves will be easily reusable on Docker anyway.

I considered using tox, but it looked like overkill (and overlapping with Travis). I considered using Python's own unittest module but IMHO it's a lot of code for little added value.

Still not addressed:

• /bin/env -> /usr/bin/env (?)
• Using the exact same Python dot versions as in prod (not supported by Travis, I don't think it's worth the effort given that the Docker image will provide a well-controlled environment)
• Use P5 data (unsure, see discussion above)
• Test other scripts than anaUltraScurve.py (will start working on it once the principle is :+1:'ed, we might as well have one PR/tested script)

[jsturdy]

I tracked down a couple of issues with the docker container that was being used, and have managed to get things looking pretty much as expected from within the travis environment. If you can let me know what you would need to add to the container to complete your tests (unless you just want to run the installation commands in your travis script), I can update the container.

• In the docker image you can just use yum install root\* to get the version of root provided with slc6/cc7
• Extra python dependencies should be handled only through a requirements.txt or a requirements-dev/prod/test.txt and pip
• It would be best to have the test data and directories set up in the container itself, or, at the very least, pulled via wget into the travis build, rather than stored in the repo
• While adding the test script (there should really be a single test driver to test all aspects of the package, which was why I had suggested tox) it would be good to add the test coverage (coveralls, codecov, etc., you can also look at my integrations branch for some testing I had been setting up, though not yet operational)

One other thing that would be optimal would be to separate the unit testing from the CI/CD setup, such that the unit tests can be run independently of the CI/CD, and will then easily plug into a functional CI/CD framework (there will be two, by virtue of the fact that the gitlab mirrors are using gitlab-ci, while the github repositories will use travis)

While moving along these lines, the best way forward will be for you to regularly do a git rebase to the mainline branch, which should be straightforward as long as you're not modifying any of the code unrelated to the travis aspects (which this PR shouldn't be doing anyway...)

[jsturdy]

closing as "On Hold"