search

LSSTDESC / Monitor

Extract light curves for time-variable cosmological objects
BSD 3-Clause "New" or "Revised" License
3 stars 1 forks source link

We need a workaround for the sncosmo/astropy incompatibilities in the conda distribution of lsst-sims #48

Closed jchiang87 closed 8 years ago

jchiang87 commented 8 years ago

This problem is causing our builds to fail at travis-ci, e.g.,

https://travis-ci.org/DarkEnergyScienceCollaboration/Monitor/builds/147903861

@heather999 I think you were able to roll back astropy to a previous version for the SLAC installation of v12_0 so that it would work with the sncosmo in the conda distribution. If so, then I'd like to update the .travis.yml install script to do the same. What's the procedure for that?

@danielsf Any chance of getting a new conda dist of lsst-sims that doesn't suffer from this issue soon-ish?

danielsf commented 8 years ago

Remind me what the problem is? Is it that sncosmo 1.2 does not work with astropy 1.2?

jchiang87 commented 8 years ago

Yes, sncosmo 1.2 needs astropy 1.1.x . Alternatively, sncosmo 1.3 (created to fix this problem) will work with astropy 1.2. So, changing one or the other accordingly would fix things.

jchiang87 commented 8 years ago

Here's the relevant issue at the sncosmo repo:

https://github.com/sncosmo/sncosmo/issues/142

danielsf commented 8 years ago

Let me see if we have an official plan for a release. If not, I can start the ball rolling today.

heather999 commented 8 years ago

I thought there was also a problem using astropy 1.2 with sims_utils (at least the version released for use with the Stack v12_0), which disappeared when we downgraded to astropy 1.1.1.. I actually went with astropy 1.1.2 as I assumed that was enough, @danielsf https://community.lsst.org/t/how-to-downgrade-astropy-in-conda-install-of-v12-0/910/9

How I did that @jchiang87.. I did the typical vanilla conda installation of v12_0: conda create --prefix "myPathToTheNewCondaEnv" --channel http://conda.lsst.codes/stack python=2 lsst-distrib source activate "myPathToTheNewCondaEnv" conda install --channel http://conda.lsst.codes/stack lsst-sims and then: conda install astropy=1.1.2

For travis-ci, I suspect you dont' want to use --prefix, but will be installing a new "lsst" environment as part of the local anaconda installation.

danielsf commented 8 years ago

Is it absolutely necessary to use astropy 1.2? I ask because the relationship between LSST and astropy is still very tenuous (we recently had an episode in which our Jenkins slaves were installing astropy 1.2 even though we thought they were installing astropy 1.1.1)

It is not clear that we have any official policy or means of controlling what version of astropy gets installed with the stack.

danielsf commented 8 years ago

@heather999 is correct: sims_utils unit tests should also fail with astropy 1.2

jchiang87 commented 8 years ago

From what I can tell, there's no reason that I can see to use astropy 1.2 at this time. The only reason I could see would be if someone really needed to use sncosmo 1.3.

danielsf commented 8 years ago

So are you saying that sncosmo 1.3 is not backwards compatible with astropy 1.1.1 (we are just trying to figure out how you got astropy 1.2, since we think our conda build is hard-coded to deliver astropy 1.1.1)

jchiang87 commented 8 years ago

I don't know the details of the changes in sncosmo 1.3, but given the nature of the incompatibilities of sncosmo 1.2 with astropy 1.2, I would be surprised if they went to the effort to make sure sncosmo 1.3 was backwards compatible with astropy 1.1.x. @rbiswas4 may be able to comment.

In any case, my understanding and the breakage I've seen seem to indicate that the v12_0 conda distribution is installing astropy 1.2.

heather999 commented 8 years ago

@jchiang87 can correct me, but I believe we got astropy 1.2 from the conda installation of v12_0 lsst-distrib. Going back through my logs.. for the lsst-distrib installation I see: The following NEW packages will be INSTALLED:

astropy:                          1.2.1-np111py27_0

Then I do an install of lsst-sims in that same conda environment where it adds: The following NEW packages will be INSTALLED:

alabaster:                     0.7.8-py27_0                                                                                                                     
astropy-helpers:               1.1.1-0                 http://conda.lsst.codes/stack (soft-link)

So it looks to me that astropy 1.2 satisfies the lsst-sims requirements, but perhaps it would have installed astropy 1.1.1 if I hadn't installed lsst-distrib first.. How is the astropy requirement specified for lsst-sims? Is it strictly requiring astropy 1.1.1?

danielsf commented 8 years ago

I just had a long conversation with Lynne about this. We're still not sure how astropy 1.2 got in there. One thing she noticed: you should be installing from http://conda.lsst.codes/sims not http://conda.lsst.codes/stack, since the sims channel is where all of the up-to-date sims releases will be published. We also have lsst_apps on the sims channel. We do not have lsst_distrib on the sims_channel (but installing lsst_sims and lsst_apps should get you everything you want).

I will try a fresh install from lsst_sims and see what version of astropy I get. Maybe the stack channel has more lax requirements on astropy.

heather999 commented 8 years ago

@danielsf That very well might be the answer.. in which case I think the doc here, needs some modification, under 2. Install Science Pipelines in a Conda Environment https://pipelines.lsst.io/install/conda.html I suspect I need to poke Lynne and Jonathan Sick about that bit.. I'll try via community. Otherwise I'll trip over this again in the future :)

danielsf commented 8 years ago

There is still a lot of confusion about where sims and DM documentation should live.

We are trying to keep the most up-to-date sims install information here

https://confluence.lsstcorp.org/display/SIM/Catalogs+and+MAF

which you can get to from

http://lsst.org/scientists/simulations

and clicking on either the CatSim or the MAF windows

Sorry for the confusion

rbiswas4 commented 8 years ago

There are now two conda installs being published. DM publishes the one @heather999 pointed to. I believe we should be using https://conda.lsst.codes/sims (I had pointed this out in a thread before), if we want the sims versions to be compatible with the DM version, aside from releases which are typically once or twice a year.

danielsf commented 8 years ago

I just did a fresh install of lsst_sims and lsst_apps from the sims channel. I get astropy 1.1.2

danielsf commented 8 years ago

trying the same install from the stack channel tries to bring in astropy-1.2.1, as @heather999 observed, so that is the problem. We will investigate why the two channels have different constraints on astropy. As always: sorry for the confusion.

jchiang87 commented 8 years ago

I set the channel in the install script to https://conda.lsst.codes/sims and the build is fixed:

https://travis-ci.org/DarkEnergyScienceCollaboration/Monitor/builds/148114144

Thanks everyone! I'll close this issue via the pull-request.