have travis try to use wheels

eteq commented 11 years ago

Right now travis builds for affiliated packages have to build all dependencies. With astropy/astropy#1029, we are using the "wheel" format to store binaries of scipy, numpy, and Cython, which speeds up the tests by more than 2x. We should change the affiliated package template's .travis.yml to mirror this approach.

Two things needed first, though:

[ ] First, though, we should figure out where we are going to store the wheels on a long-term basis. One possibility might be to use something like "wheels.astropy.org" and have that redirect to some place temporary for now (where the core package wheels are) and move it around as we get better servers.
[ ] We may also want to add a wheel for astropy itself for use by the affiliated packages. This could be tricky for packages that want both stable and "latest" dev versions of astropy, but the approach from astropy/astropy#1029 was designed with this in mind - it will try to download a wheel, and if it doesn't find it it will fall back on the source code build.

astrofrog commented 11 years ago

I had the same thought today about using a sub-domain like wheels.astropy.org to alleviate the long-term storage issue - i think that would work.

cdeil commented 10 years ago

Status? (astropy and astroquery are already using wheels from http://www.mpia.de/~robitaille/python/wheels)

astrofrog commented 10 years ago

First I think we may want to upgrade the way Astropy deals with it based on code @Cadair has developed. I'll open an issue for this.

Cadair commented 10 years ago

Hello,

I have been working on this for SunPy. I came to the conclusion that the easiest way to build wheels was on Travis so I created this repo: https://github.com/Cadair/travis_wheels that builds all the dependencies for SunPy and AstroPy and stores them here: http://sunpy.cadair.com/wheelhouse/.

I am happy to share this wheelhouse and accept PR's to the build repo for extra packages if people need them. There will need to be some restructuring of the build script if more packages need adding as the total build time is close to the Travis CI limit of 50 minutes.

Also I found a better solution to the installing packages by pip using wheels. pip can trawl a html file and use it as a package index so you can point pip to a wheelhouse by doing: --find-links http://yourdomian.com/wheelhouse/index.html --use-wheel which you can see an example of in the SunPy .travis.yml file: https://github.com/sunpy/sunpy/blob/staging/.travis.yml HTH Stuart

cdeil commented 10 years ago

@Cadair If you find that approach of creating the wheel on travis-ci problematic for some reason, another simple way it to create locally (or in the cloud, I guess) a VM with the same Ubuntu version as travis-ci and build the wheels there. This is what I did for the ROOT library which takes too long to build for travis-ci and it works.

Cadair commented 10 years ago

@cdeil Yes, that is an option. However, I thought that building the wheels in a way where people can see and change it, while making the code open was a good idea ;)

I can see if I can get ROOT working in the repo if you wish, I have a few ideas to get around the total build limit ;)

Cadair commented 10 years ago

Also you should be able to use the wheels to expand your number of Travis configs, as you no doubt already know.

astrofrog commented 10 years ago

@Cadair @cdeil - I've opened a pull request on the core package first so we can work on this:

https://github.com/astropy/astropy/pull/1355

Once this works fine, we can port over these changes to the package template.

astrofrog commented 10 years ago

I'll start working on this. Note to self: also update minimum sphinx version to 0.2b2

cdeil commented 10 years ago

@astrofrog Can you update the package-template .travis.yml to use wheels?

I've got an example in the sphinx build that uses scipy and thus fails with the current .travis.yml file: https://travis-ci.org/gammapy/gammapy/jobs/18529539#L974

astrofrog commented 10 years ago

@cdeil - I'm on vacation for a week so won't be able to do it. But I agree this is a good idea. Could you open a pull request for this? Alternatively, you could just do it in your affiliated package but not add it to the template.

cdeil commented 10 years ago

@astrofrog If you can show us how to use wheels for affiliated packages here in the package-template, that would be great!

weaverba137 commented 10 years ago

Yet another wheels-related ticket that lots of people are waiting on.

embray commented 10 years ago

IIRC I believe there might be movement away from that soon as Travis is getting better support for dependency caching. I'm not exactly sure what the last word was on that though. In the meantime there's nothing stopping anyone from updating their own .travis.yml to try pip installing from the wheel links.

weaverba137 commented 10 years ago

You're saying track the Travis configuration of astropy/astropy? I can do that, but it kind of diminishes the importance of package-template.

cdeil commented 10 years ago

@embray Should we wait for https://github.com/astropy/package-template/pull/46 to be merged before updating the .travis.yml for affiliated packages? Or is the .travis.yml file independent of these changes?

embray commented 10 years ago

No, #46 doesn't really have any impact on this that I'm aware of.

cdeil commented 10 years ago

@mwcraig is implementing this in #44.

astrofrog commented 10 years ago

Closed by #44

astropy / package-template

have travis try to use wheels #24