Decide strategy with respect to -dev releases (via labels or otherwise) over conda

[davidlmobley]

So, I realize OS X builds via travis are currently broken ( #537 ), but - could someone explain to me how they are normally (when not broken) generated via Travis-CI (?) (i.e. for openmoltools)?

I'm trying to figure out how to get Travis to generate and upload builds to anaconda.org for my package, modeling it after what openmoltools is doing in https://github.com/choderalab/openmoltools/blob/master/.travis.yml and https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh, but this actually only seems to generate linux builds via Travis-CI (yet I can conda install openmoltools on OS X) so I'm confused about where the OS X builds are coming from. I'm guessing the answer in this case is "not from travis", but if so, does someone have a working example of generating OS X builds from travis that I could consult?

Thanks.

[rmcgibbo]

Both the Linux and OS X packages on anaconda.org/omnia are built using the Travis CI associated with this repository. The OS X build process is actually simpler than the linux build, because there's no docker. On OS X, Travis's VMs run starting from this code in .travis.yml, which executes this script, which triggers a build of any recipe in this recipe which has been updated.

Neither https://github.com/choderalab/openmoltools/blob/master/.travis.yml nor https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh have anything to do with uploading compiled binaries to anaconda.org for conda install openmmtools.

Line 24 in https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh does upload a package to anaconda.org, but that package is called "openmmtools-dev", not "openmmtools" and is never built on OS X (because https://github.com/choderalab/openmoltools/blob/master/.travis.yml doesn't tell travis to use OS X -- it's linux only).

[davidlmobley]

Sorry, I'm not to the omnia stage yet. I was just trying to figure out how to build my own dev binaries for upload to anaconda.

Thanks for the pointer to the builds here though; I'll see if those recipes help me.

On Fri, Jun 10, 2016, 6:15 PM Robert T. McGibbon notifications@github.com wrote:

Both the Linux and OS X packages on anaconda.org/omnia are built using the Travis CI associated with this repository. The OS X build process is actually simpler than the linux build, because there's no docker. On OS X, Travis's VMs run starting from this code in .travis.yml https://github.com/omnia-md/conda-recipes/blob/master/.travis.yml#L36-L39, which executes this script https://github.com/omnia-md/conda-recipes/blob/master/devtools/osx-build.sh, which triggers a build of any recipe in this recipe which has been updated.

Neither https://github.com/choderalab/openmoltools/blob/master/.travis.yml nor https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh have anything to do with uploading compiled binaries to anaconda.org for conda install openmmtools.

Line 24 in https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh does upload a package to anaconda.org, but that package is called "openmmtools-dev", not "openmmtools" and is never built on OS X (because https://github.com/choderalab/openmoltools/blob/master/.travis.yml doesn't tell travis to use OS X -- it's linux only).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225328327, or mute the thread https://github.com/notifications/unsubscribe/AGzUYTQHa1X-lPBbwOYks4qltEYz-Hsjks5qKgwfgaJpZM4IzbwB .

[rmcgibbo]

I think these -dev binaries are a bad idea, which is one of the reasons that they haven't been widely used.

IMO conda is a useful tool for distributing released software to end users, which is what this repository is about. Conda is not well suited to distributing in-development snapshots to the developers themselves.

Sent from my iPhone

On Jun 10, 2016, at 9:23 PM, David L. Mobley notifications@github.com wrote:

Sorry, I'm not to the omnia stage yet. I was just trying to figure out how to build my own dev binaries for upload to anaconda.

Thanks for the pointer to the builds here though; I'll see if those recipes help me.

On Fri, Jun 10, 2016, 6:15 PM Robert T. McGibbon notifications@github.com wrote:

Both the Linux and OS X packages on anaconda.org/omnia are built using the Travis CI associated with this repository. The OS X build process is actually simpler than the linux build, because there's no docker. On OS X, Travis's VMs run starting from this code in .travis.yml https://github.com/omnia-md/conda-recipes/blob/master/.travis.yml#L36-L39, which executes this script https://github.com/omnia-md/conda-recipes/blob/master/devtools/osx-build.sh, which triggers a build of any recipe in this recipe which has been updated.

Neither https://github.com/choderalab/openmoltools/blob/master/.travis.yml nor https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh have anything to do with uploading compiled binaries to anaconda.org for conda install openmmtools.

Line 24 in https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh does upload a package to anaconda.org, but that package is called "openmmtools-dev", not "openmmtools" and is never built on OS X (because https://github.com/choderalab/openmoltools/blob/master/.travis.yml doesn't tell travis to use OS X -- it's linux only).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225328327, or mute the thread https://github.com/notifications/unsubscribe/AGzUYTQHa1X-lPBbwOYks4qltEYz-Hsjks5qKgwfgaJpZM4IzbwB .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[davidlmobley]

To clarify -in this case really all I was trying to do was figure out how the whole system works by uploading to my own conda channel before bothering all of you to try and get it into omnia. Then I was going to delete. I just felt I needed to go through all of it once so I'd understand what was really going on.

I should have made more clear that this wasn't an omnia question per se as much as a "how do people developing omnia do this" question.

Thanks. I think the clarification you provided helped.

On Fri, Jun 10, 2016, 6:29 PM Robert T. McGibbon notifications@github.com wrote:

I think these -dev binaries are a bad idea, which is one of the reasons that they haven't been widely used.

IMO conda is a useful tool for distributing released software to end users, which is what this repository is about. Conda is not well suited to distributing in-development snapshots to the developers themselves.

Sent from my iPhone

On Jun 10, 2016, at 9:23 PM, David L. Mobley notifications@github.com wrote:

Sorry, I'm not to the omnia stage yet. I was just trying to figure out how to build my own dev binaries for upload to anaconda.

Thanks for the pointer to the builds here though; I'll see if those recipes help me.

On Fri, Jun 10, 2016, 6:15 PM Robert T. McGibbon < notifications@github.com> wrote:

Both the Linux and OS X packages on anaconda.org/omnia are built using the Travis CI associated with this repository. The OS X build process is actually simpler than the linux build, because there's no docker. On OS X, Travis's VMs run starting from this code in .travis.yml < https://github.com/omnia-md/conda-recipes/blob/master/.travis.yml#L36-L39 , which executes this script < https://github.com/omnia-md/conda-recipes/blob/master/devtools/osx-build.sh , which triggers a build of any recipe in this recipe which has been updated.

Neither https://github.com/choderalab/openmoltools/blob/master/.travis.yml nor

https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh have anything to do with uploading compiled binaries to anaconda.org for conda install openmmtools.

Line 24 in

https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh does upload a package to anaconda.org, but that package is called "openmmtools-dev", not "openmmtools" and is never built on OS X (because https://github.com/choderalab/openmoltools/blob/master/.travis.yml doesn't tell travis to use OS X -- it's linux only).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225328327 , or mute the thread < https://github.com/notifications/unsubscribe/AGzUYTQHa1X-lPBbwOYks4qltEYz-Hsjks5qKgwfgaJpZM4IzbwB

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225329124, or mute the thread https://github.com/notifications/unsubscribe/AGzUYV85uMlwgiLk43oHXaqR3lcRriT6ks5qKg98gaJpZM4IzbwB .

[rmcgibbo]

In that case, the key command you need to play with is just "conda build", and also possibly "anaconda upload".

Sent from my iPhone

On Jun 10, 2016, at 9:43 PM, David L. Mobley notifications@github.com wrote:

To clarify -in this case really all I was trying to do was figure out how the whole system works by uploading to my own conda channel before bothering all of you to try and get it into omnia. Then I was going to delete. I just felt I needed to go through all of it once so I'd understand what was really going on.

I should have made more clear that this wasn't an omnia question per se as much as a "how do people developing omnia do this" question.

Thanks. I think the clarification you provided helped.

On Fri, Jun 10, 2016, 6:29 PM Robert T. McGibbon notifications@github.com wrote:

I think these -dev binaries are a bad idea, which is one of the reasons that they haven't been widely used.

IMO conda is a useful tool for distributing released software to end users, which is what this repository is about. Conda is not well suited to distributing in-development snapshots to the developers themselves.

Sent from my iPhone

On Jun 10, 2016, at 9:23 PM, David L. Mobley notifications@github.com wrote:

Sorry, I'm not to the omnia stage yet. I was just trying to figure out how to build my own dev binaries for upload to anaconda.

Thanks for the pointer to the builds here though; I'll see if those recipes help me.

On Fri, Jun 10, 2016, 6:15 PM Robert T. McGibbon < notifications@github.com> wrote:

Both the Linux and OS X packages on anaconda.org/omnia are built using the Travis CI associated with this repository. The OS X build process is actually simpler than the linux build, because there's no docker. On OS X, Travis's VMs run starting from this code in .travis.yml < https://github.com/omnia-md/conda-recipes/blob/master/.travis.yml#L36-L39 , which executes this script < https://github.com/omnia-md/conda-recipes/blob/master/devtools/osx-build.sh , which triggers a build of any recipe in this recipe which has been updated.

Neither https://github.com/choderalab/openmoltools/blob/master/.travis.yml nor

https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh have anything to do with uploading compiled binaries to anaconda.org for conda install openmmtools.

Line 24 in

https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh does upload a package to anaconda.org, but that package is called "openmmtools-dev", not "openmmtools" and is never built on OS X (because https://github.com/choderalab/openmoltools/blob/master/.travis.yml doesn't tell travis to use OS X -- it's linux only).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225328327 , or mute the thread < https://github.com/notifications/unsubscribe/AGzUYTQHa1X-lPBbwOYks4qltEYz-Hsjks5qKgwfgaJpZM4IzbwB

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225329124, or mute the thread https://github.com/notifications/unsubscribe/AGzUYV85uMlwgiLk43oHXaqR3lcRriT6ks5qKg98gaJpZM4IzbwB .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jchodera]

@rmcgibbo: We were thinking of phasing out the -dev packages in favor of uploading a 0.0.0 package to the dev label. Do you think this is a bad idea too?

If so, any suggestions for packages that need to test against multiple dependencies on Omnia that are still evolving? Frequent point releases instead?

[rmcgibbo]

The dev "label" is slightly better than the -dev package, but yes, I still think that the best tools for developing software are git and pip install -e, not conda.

Sent from my iPhone

On Jun 10, 2016, at 10:22 PM, John Chodera notifications@github.com wrote:

@rmcgibbo: We were thinking of phasing out the -dev packages in favor of uploading a 0.0.0 package to the dev label. Do you think this is a bad idea too?

If so, any suggestions for packages that need to test against multiple dependencies on Omnia that are still evolving? Frequent point releases instead? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[davidlmobley]

Right, I got those working earlier. I was just trying to understand how the process of "travis builds it for you on several platforms and then automatically uploads it" works. I got that for linux now, so I think I'm pretty much set.

Hopefully I can just move to getting a working recipe into omnia very soon then.

Thanks.

On Fri, Jun 10, 2016 at 6:47 PM, Robert T. McGibbon < notifications@github.com> wrote:

In that case, the key command you need to play with is just "conda build", and also possibly "anaconda upload".

Sent from my iPhone

On Jun 10, 2016, at 9:43 PM, David L. Mobley notifications@github.com wrote:

To clarify -in this case really all I was trying to do was figure out how the whole system works by uploading to my own conda channel before bothering all of you to try and get it into omnia. Then I was going to delete. I just felt I needed to go through all of it once so I'd understand what was really going on.

I should have made more clear that this wasn't an omnia question per se as much as a "how do people developing omnia do this" question.

Thanks. I think the clarification you provided helped.

On Fri, Jun 10, 2016, 6:29 PM Robert T. McGibbon < notifications@github.com> wrote:

I think these -dev binaries are a bad idea, which is one of the reasons that they haven't been widely used.

IMO conda is a useful tool for distributing released software to end users, which is what this repository is about. Conda is not well suited to distributing in-development snapshots to the developers themselves.

Sent from my iPhone

On Jun 10, 2016, at 9:23 PM, David L. Mobley < notifications@github.com> wrote:

Sorry, I'm not to the omnia stage yet. I was just trying to figure out how to build my own dev binaries for upload to anaconda.

Thanks for the pointer to the builds here though; I'll see if those recipes help me.

On Fri, Jun 10, 2016, 6:15 PM Robert T. McGibbon < notifications@github.com> wrote:

Both the Linux and OS X packages on anaconda.org/omnia are built using the Travis CI associated with this repository. The OS X build process is actually simpler than the linux build, because there's no docker. On OS X, Travis's VMs run starting from this code in .travis.yml <

https://github.com/omnia-md/conda-recipes/blob/master/.travis.yml#L36-L39

,

which executes this script <

https://github.com/omnia-md/conda-recipes/blob/master/devtools/osx-build.sh

,

which triggers a build of any recipe in this recipe which has been updated.

Neither https://github.com/choderalab/openmoltools/blob/master/.travis.yml nor

https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh

have anything to do with uploading compiled binaries to anaconda.org for conda install openmmtools.

Line 24 in

https://github.com/choderalab/openmoltools/blob/master/devtools/travis-ci/after_success.sh

does upload a package to anaconda.org, but that package is called "openmmtools-dev", not "openmmtools" and is never built on OS X (because https://github.com/choderalab/openmoltools/blob/master/.travis.yml doesn't tell travis to use OS X -- it's linux only).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225328327

,

or mute the thread <

https://github.com/notifications/unsubscribe/AGzUYTQHa1X-lPBbwOYks4qltEYz-Hsjks5qKgwfgaJpZM4IzbwB

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225329124 , or mute the thread < https://github.com/notifications/unsubscribe/AGzUYV85uMlwgiLk43oHXaqR3lcRriT6ks5qKg98gaJpZM4IzbwB

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225330073, or mute the thread https://github.com/notifications/unsubscribe/AGzUYXof2wtOwWW-3E35X_TILbwCa4NGks5qKhOrgaJpZM4IzbwB .

David Mobley dmobley@gmail.com 949-385-2436

[davidlmobley]

Back to the philosophy issues a bit:

IMO conda is a useful tool for distributing released software to end users, which is what this repository is about. Conda is not well suited to distributing in-development snapshots to the developers themselves.

I do use the dev snapshots a lot in my group - for example, it's not unusual we uncover a bug in a package which we report and which gets fixed. Once it's fixed, since I have some people who are using the software in question but not actively developing, it's often easier to tell them to make sure they are using, say, parmed-dev via conda install parmed-dev (and they can deviate from this if they know what they are doing) than to have them retrieve parmed themselves; this is especially true if (as is true in some cases in my group) the students are new undergrads and are ONLY used to installing via conda.

So, my take is there's room for conda-installable development versions, however we get access to them. We certainly use these a lot, in my group.

[rmcgibbo]

I do use the dev snapshots a lot in my group ...

I'm not sure what this shows -- the fact that your students use these conda binaries is not, in and of itself, evidence that conda is a good tool for distributing development snapshots. To the contrary, the conda parmed-dev files on anaconda.org are currently clearly outdated with respect to the master branch of the repository. The conda files appear to have been last uploaded over 1 month ago, while the latest commit to github.com/parmed/parmed was 7 days ago.

On the other hand, $ pip install git+git://github.com/parmed/parmed will actually install the latest version of the parmed.

[jchodera]

Parmed-dev hasn't been updated because we had to disable the conda-dev-recipes when updates to conda-build-all broke that repo, you may recall. Fixing this is something we haven't had time to yet.

Your pip install suggestion only works if all dependencies are automatically installable via pip, right? What if some packages are only available via conda? Or what is the difference between installing some packages via their conda vs pip versions? These are not even guaranteed to be synchronized. And is this the best thing to use for testing if it isn't guaranteed that you are getting the conda versions of dependencies?

To me, the pip route sounds even WORSE than what we currently do, at least for packages that have dependencies in conda.

[rmcgibbo]

The -dev recipes are seemingly always somewhat broken or disabled though, which is my point. The pip solution actually does work.

Sent from my iPhone

On Jun 11, 2016, at 6:47 AM, John Chodera notifications@github.com wrote:

Parmed-dev hasn't been updated because we had to disable the conda-dev-recipes when updates to conda-build-all broke that repo, you may recall. Fixing this is something we haven't had time to yet.

Your pip install suggestion only works if all dependencies are automatically installable via pip, right? What if some packages are only available via conda? Or what is the difference between installing some packages via their conda vs pip versions? These are not even guaranteed to be synchronized. And is this the best thing to use for testing if it isn't guaranteed that you are getting the conda versions of dependencies?

To me, the pip route sounds even WORSE than what we currently do, at least for packages that have dependencies in conda.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jchodera]

Really?

[LSKI1497:~] choderaj% bash Miniconda2-latest-MacOSX-x86_64.sh -b -p miniconda
PREFIX=/Users/choderaj/miniconda
installing: _cache-0.0-py27_x0 ...
installing: python-2.7.11-0 ...
installing: conda-env-2.4.5-py27_0 ...
installing: openssl-1.0.2g-0 ...
installing: pycosat-0.6.1-py27_0 ...
installing: pyyaml-3.11-py27_1 ...
installing: readline-6.2-2 ...
installing: requests-2.9.1-py27_0 ...
installing: sqlite-3.9.2-0 ...
installing: tk-8.5.18-0 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-0 ...
installing: conda-4.0.5-py27_0 ...
installing: pycrypto-2.6.1-py27_0 ...
installing: pip-8.1.1-py27_1 ...
installing: wheel-0.29.0-py27_0 ...
installing: setuptools-20.3-py27_0 ...
Python 2.7.11 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
[LSKI1497:~] choderaj% pip install git+git://github.com/parmed/parmed
Collecting git+git://github.com/parmed/parmed
  Cloning git://github.com/parmed/parmed to /var/folders/m1/wl3ptzqx2csf2r_q2bfy78n0h7kcfv/T/pip-YevjJn-build
Installing collected packages: ParmEd
  Running setup.py install for ParmEd ... done
Successfully installed ParmEd-2.5.1+49.ge9752f8
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[LSKI1497:~] choderaj% python
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:57:58) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import parmed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/__init__.py", line 22, in <module>
    from parmed import unit, utils
  File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/unit/__init__.py", line 27, in <module>
    from parmed.unit.unit import Unit, is_unit
  File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/unit/unit.py", line 35, in <module>
    from parmed.utils.six import iterkeys
  File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/utils/__init__.py", line 3, in <module>
    from parmed.utils.pairlist import find_atom_pairs
  File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/utils/pairlist.py", line 5, in <module>
    import numpy as np
ImportError: No module named numpy
>>> 

Looks like this doesn't work at all.

And how would it pull in openmm as a dependency?

I think we may have different definitions of "work".

[jchodera]

I think the pip scheme could be made to work if we reengineered a new development package dependency installation scheme around it. But is it worth the investment in that, or in something based around dev labels and conda? I'm not sure which plan is best.

[rmcgibbo]

When you're developing software, you don't have numpy installed? We're not talking about distributing software to end users, we're talking about developers by assumption.

Sent from my iPhone

On Jun 11, 2016, at 12:42 PM, John Chodera notifications@github.com wrote:

Really?

[LSKI1497:~] choderaj% bash Miniconda2-latest-MacOSX-x86_64.sh -b -p miniconda PREFIX=/Users/choderaj/miniconda installing: _cache-0.0-py27_x0 ... installing: python-2.7.11-0 ... installing: conda-env-2.4.5-py27_0 ... installing: openssl-1.0.2g-0 ... installing: pycosat-0.6.1-py27_0 ... installing: pyyaml-3.11-py27_1 ... installing: readline-6.2-2 ... installing: requests-2.9.1-py27_0 ... installing: sqlite-3.9.2-0 ... installing: tk-8.5.18-0 ... installing: yaml-0.1.6-0 ... installing: zlib-1.2.8-0 ... installing: conda-4.0.5-py27_0 ... installing: pycrypto-2.6.1-py27_0 ... installing: pip-8.1.1-py27_1 ... installing: wheel-0.29.0-py27_0 ... installing: setuptools-20.3-py27_0 ... Python 2.7.11 :: Continuum Analytics, Inc. creating default environment... installation finished. [LSKI1497:~] choderaj% pip install git+git://github.com/parmed/parmed Collecting git+git://github.com/parmed/parmed Cloning git://github.com/parmed/parmed to /var/folders/m1/wl3ptzqx2csf2r_q2bfy78n0h7kcfv/T/pip-YevjJn-build Installing collected packages: ParmEd Running setup.py install for ParmEd ... done Successfully installed ParmEd-2.5.1+49.ge9752f8 You are using pip version 8.1.1, however version 8.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. [LSKI1497:~] choderaj% python Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:57:58) [GCC 4.2.1 (Apple Inc. build 5577)] on darwin Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org

import parmed Traceback (most recent call last): File "", line 1, in File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/init.py", line 22, in from parmed import unit, utils File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/unit/init.py", line 27, in from parmed.unit.unit import Unit, is_unit File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/unit/unit.py", line 35, in from parmed.utils.six import iterkeys File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/utils/init.py", line 3, in from parmed.utils.pairlist import find_atom_pairs File "/Users/choderaj/miniconda/lib/python2.7/site-packages/parmed/utils/pairlist.py", line 5, in import numpy as np ImportError: No module named numpy

Looks like this doesn't work at all.

And how would it pull in openmm as a dependency?

I think we may have different definitions of "work".

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jchodera]

Apologies for being unclear: I'm talking about automated travis testing involving upstream packages that rapidly change. The current system is mostly broken, and we're evaluating a few options for fixing this:

• Insist on frequent point releases for all packages and only use upstream releases; don't upload travis-built packages to omnia.
• Switch to having travis upload packages to the dev label, and use these packages if needed by manually specifying we need version 0.0.0 for testing
• Build some automated system around pip to install directly from github for those packages that need it, requiring constructing some sort of mix of conda and pip
• Other possibilities we may have missed?

[swails]

I should probably have ParmEd declare numpy as an explicit dependency for pip to understand, so that's probably my fault. That said, once numpy is installed, ParmEd just works. If you need parts that need OpenMM, you'll be told to install OpenMM (which you can easily do via conda).

I tend to agree with @rmcgibbo that, at least for packages that work well with pip at the source level (so not OpenMM, for instance), pip install from the head of the git tree beats the *-dev conda packages that never seem to work reliably for automated testing (and it's time consuming to even do it unreliably, too).

I've also never had problems with pip not understanding things that conda has installed, and vice versa. They will never use each other to satisfy dependencies, but they harmonize with each other just fine. So such a suggestion will not scale if you need to use lots of *-dev packages.

But if you're living on the bleeding edge for so many packages that sometimes using pip is too much of a pain, you're asking for a whole lot more problems than difficulty in resolving cross-package-manager dependencies.

So if you just need to make sure you get the latest ParmEd instead of the stable conda package, just use pip.

[mpharrigan]

conda install parmed and then pip install master branch. conda will resolve all the dependencies and you'll get the latest parmed code. substitute parmed for anything

[davidlmobley]

I'm inclined to chime in on the side of conda over pip. I've had undergrads on Windows with no programming/computing experience who can conda install openmoltools-dev and have everything just work (despite me not knowing whether this would really work on Windows) and getting them going with pip would have been much more painful.

Matthew Harrigan's solution also sounds good. But certainly pip is not as good a solution as conda for rookies (who sometimes DO need to use the latest code).

On Sat, Jun 11, 2016 at 11:56 AM, Matthew Harrigan <notifications@github.com

wrote:

conda install parmed and then pip install master branch. conda will resolve all the dependencies and you'll get the latest parmed code. substitute parmed for anything

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/omnia-md/conda-recipes/issues/539#issuecomment-225385872, or mute the thread https://github.com/notifications/unsubscribe/AGzUYdNOEcKMlLp-smj5upVr1E2GWpi1ks5qKwTVgaJpZM4IzbwB .

David Mobley dmobley@gmail.com 949-385-2436

[mpharrigan]

I'm not sure if making it easy for undergrads on Windows with no programming/computing experience to get bleeding-edge academic code running should be our priority. This use case seems much better suited to frequent releases, stable releases, or both

[rmcgibbo]

I'm inclined to chime in on the side of conda over pip. I've had undergrads on Windows with no programming/computing experience who can conda install openmoltools-dev

Are you sure? There are no Windows binaries for openmm-dev on anaconda.org.

Let me just run through the my memory of the sequence of events. When we started setting up conda-based buid infrastructure and continuous integration, IIRC a few of the PG projects like MDTraj and MSMBuilder changed their travis scripts to upload built binaries after testing to anaconda.org (then called binstar.org). John then began to use this pattern for some of his group's software as well. This was all done using travis-ci's linux build infrastructure. There were never any -dev packages for Windows or OS X. If I remember correctly, the packages were mostly intended to be used inside the travis-ci test scripts for other downstream packages that depended on MDTraj, MSMBuilder, and the other packages that produced these nightly build artifacts.

These binaries had a lot of problems, which led Matt and I to remove them for the packages that we worked on. Some of those problems, from my perspective, were.

• The binaries were not cross-platform. They also didn't work on many linux distributions in common use like CentOS 5 and CentOS 6, which led to frustration. Compiling linux binaries that work cross-distro is kind of tricky. You can read a little bit about the problems and solutions in the PEP I coauthored.
• The binaries were never versioned properly. We didn't want to use all of our storage on anaconda.org, so the version numbers were always wrong so that the packages would overwrite one another when uploaded.
• The machinery to build and upload these packages was seemingly always broken. This seems to still be the case, given that the current parmed-dev binaries are outdated.

John and David seem to be talking about two different use cases in this thread.

1. How to configure automated testing for packages with rapidly-moving upstream dependencies?
   • This doesn't require any coordination between parties. My recommendation is that any mixture of conda install, pip install, git clone ... && python setup.py install is just fine, dependening on how you want to structure your scripts and tests, and you want to pull from conda/pypi vs. from github.
2. How to distribute software to users like undergraduates?
   • If a large fraction of your end users have a dire need for the tip of the master branch as opposed to the latest release of the software, that's a good signal you're probably releasing too infrequently in my opinion.