Devtools

[maxentile]

Standard structure, following John's comment: https://github.com/choderalab/msm-pipeline/pull/9#issuecomment-234250613

[jchodera]

Great start, but you'll need to update the .travis.yml to use the testing framework too:

Here's an example: https://github.com/choderalab/alchemy/blob/master/.travis.yml

We probably want to make your pipeline script an "entry point", where running a script command (like msm-pipeline) that gets installed into ~/miniconda/bin/ will call into an entry point like msmpipeline/cli.py into a function like run_pipeline() or something. More info on entry points here:

• http://python-packaging.readthedocs.io/en/latest/command-line-scripts.html
• https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/

[jchodera]

Why not use conda to install omnia dependencies? Are you using things that aren't available through omnia?

[jchodera]

Sorry for all the travis pains.

We've tried to minimize these by sticking to a fairly standard structure for most of our projects.

For example, in smarty, we have this block:

language: c
sudo: false

branches:
  only:
    - master

install:
  - source devtools/travis-ci/install.sh
  - export PYTHONUNBUFFERED=true

script:
  # Add omnia channel
  - conda config --add channels ${ORGNAME}
  # build the recipe
  - conda build devtools/conda-recipe
  # Test the local installation
  - source activate _test
  # This is a temporary workaround for the conda issues.
  - conda install --yes nose nose-timer numpy networkx lxml openmm
  #- pip install .
  # Install OpenEye toolkit
  - pip install $OPENEYE_CHANNEL openeye-toolkits && python -c "import openeye; print(openeye.__version__)"
  # Run tests
  #- cd devtools && nosetests $PACKAGENAME --nocapture --verbosity=2 --with-doctest --with-timer -a "\!slow" && cd ..
  - python --version
  - cd devtools && nosetests -vv --nocapture --with-timer $PACKAGENAME && cd ..

env:
  matrix:
    - python=2.7  CONDA_PY=27
    - python=3.4  CONDA_PY=34
    - python=3.5  CONDA_PY=35

  global:
    - ORGNAME="omnia"
    - PACKAGENAME="smarty"

The install: section installs Miniconda, while the script: section builds the conda dev recipe in devtools/conda-recipe. If you have a test: section in the conda dev recipe, it also sets up a conda environment called _test that you can also install other stuff into (as we do here, with the OpenEye tools). You can test your script inside the conda-recipe/meta.yaml, as we do here for YANK:

test:
  requires:
    - nose
    - nose-timer
  imports:
    - yank
  commands:
    - yank --help
    - yank selftest

[maxentile]

Thanks, John!

And sorry to all subscribers for the deluge of Github notification spam the last few hours -- most of this complication was self-inflicted. I first directly copied this structure from FAHMunge, and then I tried to simplify / remove bits of the "boilerplate" that didn't seem needed here (e.g. pushing docs to S3) and inadvertently broke a bunch of things, which I think are now mostly fixed:

• setup.py -- Removed almost all of this, retaining minimal package information and the call to versioneer.get_version()
• devtools/travis-ci/-- The only thing I think I need to retain from these scripts is to fetch and install Miniconda -- I replaced this with a few lines in .travis.yml.
• devtools/conda-recipe/ -- Moved the build.sh one-liner into meta.yaml. Also didn't duplicate the readme.

Remaining minutia:

1. versioneer.py sets the version in setup.py, but the version is also hardcoded in meta.yaml. To-do: use versioneer's version inside the meta.yaml file.
2. Set the matplotlib backend correctly. I think I've made sure matplotlib.use('Agg') is before every use of matplotlib, but I keep getting an "Invalid DISPLAY variable error" accompanied by this warning: "This call to matplotlib.use() has no effect because the backend has already been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time."

[jchodera]

The reason we avoid mucking about with the stock boilerplate is to avoid the technical debt of having everything be different. By spending a day of self-inflicted pain as a direct result of mucking with it, I hope you understand now why we don't do this. :)

For (1), you can't use the output of versioneer in meta.yaml, to my knowledge. @jhprinz has looked into this, but there may bit be a good solution yet.

For (2), I figured this out in some of my scripts --- trying to figure out which ones.

[jchodera]

I think this idiom should work for the matplotlib issue:

import matplotlib as mpl
mpl.use('Agg')
import seaborn as sns

from matplotlib.backends.backend_pdf import PdfPages
import matplotlib.pyplot as plt

[maxentile]

Thanks John! I'll try that now.

[maxentile]

Will sort out the Travis error later. Merging.