`conda install` installing wrong version on numpy or theano

[jsalvatier]

The recent conda install change seems to install the wrong version of numpy (should be >1.8). This is making some of the examples fail to run because inplace_increment is missing from theano.

https://github.com/pymc-devs/pymc/blob/master/.travis.yml#L18

from https://s3.amazonaws.com/archive.travis-ci.org/jobs/18145721/log.txt :

ERROR: pymc.tests.test_examples.test_examples0('pymc.examples.GHME_2013',)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/pymc-devs/pymc/pymc/tests/test_examples.py", line 13, in check_example
    example = __import__(example_name, fromlist='dummy')
  File "/home/travis/build/pymc-devs/pymc/pymc/examples/GHME_2013.py", line 81, in <module>
    s = find_MAP( vars=[sd, y])
  File "/home/travis/build/pymc-devs/pymc/pymc/tuning/starting.py", line 69, in find_MAP
    dlogp = bij.mapf(model.fastdlogp(vars))
  File "/home/travis/build/pymc-devs/pymc/pymc/model.py", line 74, in fastdlogp
    return self.model.fastfn(gradient(self.logpt, vars))
  File "/home/travis/build/pymc-devs/pymc/pymc/memoize.py", line 14, in memoizer
    cache[key] = obj(*args, **kwargs)
  File "/home/travis/build/pymc-devs/pymc/pymc/theanof.py", line 51, in gradient
    return t.concatenate([gradient1(f, v) for v in vars], axis=0)
  File "/home/travis/build/pymc-devs/pymc/pymc/theanof.py", line 43, in gradient1
    return t.flatten(t.grad(f, v, disconnected_inputs='warn'))
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/gradient.py", line 528, in grad
    grad_dict, wrt, cost_name)
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/gradient.py", line 1103, in _populate_grad_dict
    rval = [access_grad_cache(elem) for elem in wrt]
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/gradient.py", line 1063, in access_grad_cache
    term = access_term_cache(node)[idx]
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/gradient.py", line 924, in access_term_cache
    input_grads = node.op.grad(inputs, new_output_grads)
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/tensor/subtensor.py", line 1848, in grad
    *rest)] + \
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/gof/op.py", line 411, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/home/travis/anaconda/envs/testenv/lib/python2.7/site-packages/theano/tensor/subtensor.py", line 1919, in make_node
    'Could not import inplace_increment, so some advanced '
NotImplementedError: Could not import inplace_increment, so some advanced indexing features are disabled. They will be available if you update NumPy to version 1.8 or later, or to the latest development version. You may need to clear the cache (theano-cache clear) afterwards.

[twiecki]

Damn, yes, I think they haven't upgraded their numpy yet...

[jsalvatier]

What options do we have?

1. Revert the conda changes
2. Skip the tests that stall or fail
3. Force install a different version of numpy?
4. ?

[kyleam]

If conda makes other things easier, I prefer 3. Will it work if numpy is installed with pip before the conda command?

diff --git i/.travis.yml w/.travis.yml
index 1662dc3..8deac5e 100644
--- i/.travis.yml
+++ w/.travis.yml
@@ -15,7 +15,8 @@ before_install:
 install:
   - conda create -n testenv --yes pip python=$TRAVIS_PYTHON_VERSION
   - source activate testenv
-  - conda install --yes ipython==1.1.0 pyzmq numpy scipy nose matplotlib pandas Cython scikits.sparse patsy statsmodels
+  - pip install git+https://github.com/numpy/numpy.git
+  - conda install --yes ipython==1.1.0 pyzmq scipy nose matplotlib pandas Cython scikits.sparse patsy statsmodels
   - pip install --no-deps numdifftools
   - pip install git+https://github.com/Theano/Theano.git
   - python setup.py build_ext --inplace

[twiecki]

I do want to keep conda as well but there might be issues as numpy seems to be a core dependency other packages are linked against. @asmeurer might know if it's possible.

[twiecki]

@asmeurer: "Actually, that's the timeline for the Anaconda release. NumPy 1.8 should go in the repos this weekend, unless we hit any snags before then. "

I suggest we wait.

[asmeurer]

What you suggest probably won't be an issue. I don't think the NumPy ABI changed this release, so I think even if a package was built against 1.7, it should still work with 1.8. But if waiting is an option you should do that as it will be easier.

Sorry for taking so long to update NumPy to 1.8. We usually update packages faster, but NumPy has so many dependencies that it required more extensive testing.

[twiecki]

@kyleam In that case I think the pip install should be after the conda install, no?

[kyleam]

In that case I think the pip install should be after the conda install, no?

I was thinking that numpy should be installed beforehand so that conda doesn't pull in numpy when it installs packages that depend on it. I haven't used conda much, though, so I was just guessing.

I agree that it makes sense to just wait at this point.

[asmeurer]

Conda doesn't check what is already installed from anything else like pip does. It uses its own metadata system.

[twiecki]

OK, works now.