Continuous Integration with Travis-CI

[zbeekman]

I'm working on getting a unit testing and continuous integration setup up and running. Please provide some feedback on desired features by adding them to the list in this top level comment. To discuss proposals, please create comments below.

Desired functionality (please add or comment, @jlinford @srinathv and any other fellow ParaToolsInc developers):

• [x] Run all unit tests on all pushed branches and proposed pull requests
• [x] Run pylint (version >= 1.4) and require a score >= 9 against the whole codebase OR
• [x] Run pylint (version >= 1.4) and require a score >= 9 only on changed Python files
• [x] Automatically regenerate and deploy Sphinx generated documentation on all changes pushed to master
  • @jlinford I'll need commit rights to do this since I'll have to embed a GH token in a private variable on travis-ci.org to allow the CI infra to push to gh-pages
  • This assumes the current documentation is generated only from the master branch. If this is not the case, a more complicated solution may be possible
  • Since there is only one gh-pages branch, we don't want to have each branch clobbering the documentation on gh-pages and we certainly don't want pull requests (PRs) from writing new documentation because they are only proposed changes and may be opened by anyone.
• [x] Report on unit test coverage with http://codecov.io (or http://coveralls.io but my experience has led me to greatly prefer the functionality and usability of codecov.io)
• [x] Add badges to README.md to report on:
  • [x] Build status
  • [x] Code coverage statistics
  • [x] License
  • [ ] PyPi version (eventually)
• [x] Programmatically enforce coding standards such as:
  • [x] Tau Style guidelines (pylint as described above) from: http://paratoolsinc.github.io/taucmdr/docs/developer/style.html
  • [ ] Design principles from http://paratoolsinc.github.io/taucmdr/docs/developer/design.html (appears to be a WIP now....)
  • [x] Documentation standards enforcement
  • Can this be done through pylint? Some other tool?
• Test Matrix:
  • OSes:
  • [x] osx
  • [x] linux
  • Python versions
  • [x] 2.7.7
  • [x] 2.7.x, where x > 7 ?
  • [ ] 3.x ?
• [ ] Other quality assurance tools? (All free for open source)
  • [ ] landscape Python centric linter/smell detector. I have never used it but it looks very promising
  • [ ] quantifiedcode python specific static analysis, possibly using pylint, with automated suggested fixes via emailed patches or PRs
  • [x] codeclimate uses various engines (PEP8, Radon) to find issues, and analyze cyclomatic complexity (MacCabe's) Halstead metrics etc.
  • [ ] Codacy comprehensive suite of style and static analysis/linting tools I have never used this, it seems to have a lot of fine grained customizations and a number of useful dashboards.

Of all these "Other quality assurance tools, landscape or quantified code seem like the most attractive, and hopefully they will respect the .pylintrc specifications. Code climate is attractive for providing additional metrics. Thoughts?

[zbeekman]

It looks like someone has gone through my list to indicate desired features :+1: Just please note, anyone with comment edit rights for ParaTools Inc. can check and uncheck those boxes, and no history is recorded.

A few notes on my progress so far:

• I had gotten the 2.7 patch confused with minor version in my above comment; there is no 2.8 or 2.9 yet, but there are different versions of 2.7. Is it worth while to test a number of these, or just use the Travis-CI default, 2.7.9?
• Unit tests error with Python 3.3 and build.sh fails. This seems to indicate to me that Python 3.x is not currently supported. Are there plans to support 3.x in the future?
• Getting the python environment correct on OS X is non-trivial... The pyca/cryptography repo has some good example code
• Pylint returns scores less than 9 for a number of files. If we setup a pylint score >= 9 requirement on the entire repository, then the build status will be "failing" until all code meets the pylint score requirement. On the other hand, if we enforce a pylint score on only files edited in the commit range being pushed, that will enforce an incremental payback of the technical debt found by pylint, where even if you edit just a single line of the file, you will be forced to address all pylint errors before merging. This will make the build status more meaningful and incentivize people to fix pylint problems. I'm leaning towards this approach, at least for now, until we can fix all the pylint errors.

[jlinford]

I had more but United Airlines dropped my wifi mid comment :-(

• No plans for Python 3 support right now, we'll insist on Python 2.7.x until Python 3 is widely available.
• Only master branch should be tested and linted automatically. Email alerts for broken code or lint < 9 on master would be great. I often leave broken code on my branch until I know how best to fix it. Developers should police their own branches with pylint until they feel safe merging to master.
• You should have commit rights on all branches, please try ssh if https isn't working.
• All the heavy lifting is done by TAU in native code; Python runs the UI, so no plans for PyPy yet. If we're not getting the performance we need we could look at it later.
• Pylint can enforce some documentation standards. I would love to see a tool identify undocumented class members, or documented members that no longer exist.
• Autoregen docs on push would be great. "make update-github-pages" in docs does it manually.
• QA via codeclimate etc. nice but not a priority. My experience with cyclomatic complexity is that it tells me when my code is complex, or not complex, in unactionable ways. "Yes, my code is complex, the problem is complex...... ANNNNNDD??? :-)"
• master branch has a pylint score of 9.64/10 at this moment. Are you running with the pylintrc file? e.g. pylint --rcfile=pylintrc packages/tau

[zbeekman]

I had more but United Airlines dropped my wifi mid comment :-(

Oh no!

No plans for Python 3 support right now, we'll insist on Python 2.7.x until Python 3 is widely available.

:+1:

Only master branch should be tested and linted automatically. Email alerts for broken code or lint < 9 on master would be great. I often leave broken code on my branch until I know how best to fix it. Developers should police their own branches with pylint until they feel safe merging to master.

We could certainly limit the testing to master but I suggest we just test everything. Here's why:

The Travis-CI tests run asynchronously, for free, in parallel on someone else's hardware. There are no real downsides to running the tests, even if there are branches being pushed that will intentionally fail the CI tests. Part of the beauty of CI is that you can test and lint proposed changes (pull requests) as well as production code, and WIP code. AFAICT, if your private branch is failing Travis-CI will not complain about this to anyone other than the committers in the commit range being tested. Further more, you can control notification settings in your own branch through editing the .travis.yml config file if you really wanted to squelch the warnings/etc. In the github protected branches settings you can configure which branches, if any are required to have PRs pass Travis-CI tests before merging (and admins can always override this). We can limit the README build status badge to just the master branch. Default email notifications are: on failure: always, on success: change.

TL;DR: I suggest we just test everything, at worse you'll get a few expected emails about failing branches when you push to github. Furthermore, if you are doing TDD you'll know when your branch might be kosher to merge.

You should have commit rights on all branches, please try ssh if https isn't working.

Ok, great... just wondering how you set this up... Did you add me to the ParaTools organization? (Or is paratoolsinc a user?)

All the heavy lifting is done by TAU in native code; Python runs the UI, so no plans for PyPy yet. If we're not getting the performance we need we could look at it later.

I'm a bit confused... maybe I had mentioned PyPy in the test matrix section above then subsequently deleted it. My comment about PyPi above was about the Python Package Index.

Pylint can enforce some documentation standards. I would love to see a tool identify undocumented class members, or documented members that no longer exist.

I'll put this on the "someday" list. I agree this functionality would be great. I'm just starting to flex my python muscles so it may take me some time to get up to speed and find a suitable solution here.

Autoregen docs on push would be great. "make update-github-pages" in docs does it manually.

Awesome! I'm working on this now. One slight complication is that Travis-CI checks out code in detached head so i need to tweak that Makefile appropriately to handle this because git rev-parse --abbrev-ref HEAD is not going to behave as expected when invoked on Travis-CI. I think I have a fix.

QA via codeclimate etc. nice but not a priority. My experience with cyclomatic complexity is that it tells me when my code is complex, or not complex, in unactionable ways. "Yes, my code is complex, the problem is complex...... ANNNNNDD??? :-)"

Fair. Also going on the "someday" list. Some of these tools claim to be able to make actionable suggestions to improve various code quality metrics. I have no idea how effective they are, however.

master branch has a pylint score of 9.64/10 at this moment. Are you running with the pylintrc file? e.g. pylint --rcfile=pylintrc packages/tau

OK, I think my version of pylint may be too new. This is what I get:

$ pylint --version
Warning: option profile is obsolete and it is slated for removal in Pylint 1.6.
Warning: option include-ids is obsolete and it is slated for removal in Pylint 1.6.
Warning: option symbols is obsolete and it is slated for removal in Pylint 1.6.
Warning: option comment is obsolete and it is slated for removal in Pylint 1.6.
Warning: option required-attributes is obsolete and it is slated for removal in Pylint 1.6.
Warning: option ignore-iface-methods is obsolete and it is slated for removal in Pylint 1.6.
Warning: option zope is obsolete and it is slated for removal in Pylint 1.6.
pylint 1.5.4,
astroid 1.4.4
Python 2.7.11 (default, Jan 22 2016, 08:29:18)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]
$ pylint --rcfile=pylintrc packages/tau
# [snip]
Global evaluation
-----------------
Your code has been rated at -1.64/10 (previous run: -1.64/10, +0.00)

I could probably install an older version with pip... what version are you using? Or maybe there's something in my OS X environment causing trouble?

I'm going to add a requirements.txt at the project root (or I could change the name/location if you have suggestions) to install python modules needed for development and testing. Using this I can pin pylint to an appropriate version to maintain the 9.64 score.

[zbeekman]

Actually that pylint issue looks suspiciously like a bug of some variety.

Here is where pylint is called, for example, on OS X Yosemite in job 24.2: https://travis-ci.org/zbeekman/taucmdr/jobs/114352149#L1308 Scroll to the end for the output.

[jlinford]

I'm using pylint 1.4.4. Feel free to update pylintrc if it's out of date.

We could certainly limit the testing to master but I suggest we just test everything. Here's why:

Ok, I'm sold. Test all the things!

I'm a bit confused... comment about PyPi above

Nope, that confusion was all mine, I read the wrong vowel. TAU Commander in PyPi would be cool, especially if it helps draw users to www.taucommander.com (not yet active) where we will sell them services etc.

One slight complication is that Travis-CI checks out code in detached head so i need to tweak that Makefile appropriately to handle this because...

You're very welcome (invited even) to improve on whatever we have here. That makefile is not sacred.

[zbeekman]

I'm using pylint 1.4.4. Feel free to update pylintrc if it's out of date.

I'm not sure I'd know where to start... I'll poke around and see if switching to 1.4.4 makes things behave as expected. Otherwise I may ask for a hand at some point (wednesday maybe) figuring out why I can't reproduce the pylint scores.

Ok, I'm sold. Test all the things!

:smile:

Nope, that confusion was all mine, I read the wrong vowel. TAU Commander in PyPi would be cool, especially if it helps draw users to www.taucommander.com (not yet active) where we will sell them services etc.

OK, great. I'll need to learn about the pip installation process a bit more first.

You're very welcome (invited even) to improve on whatever we have here. That makefile is not sacred.

Thanks, I've been hacking at it a little bit... I need to ensure that it can still function the "old" (aka manual) way too, if possible, without overcomplicating things.

[jlinford]

I upgraded pylint on east03 to 1.5.4 and now I'm getting the same low score as seen on Travi-CI. Let's update the pylintrc file for the latest version of pylint or address the newly reported errors so we get master back above 9/10.

[zbeekman]

Sounds good. I'm still not sure that this is the entire source of my the Travis-CI problem with pylint. I confirmed that 1.4.4 was causing the same issues on Travis.