Numpy / Theano version problem? Noisemes fails selftest

[riebling]

Something must have changed I have not yet managed to track down, causing Noisemes to fail the self-test. (may be a numpy / theano update problem) Working backwards from things I've tried:

• Gist of the original error in noisemes-test.log:

  Traceback (most recent call last):
  File "SSSF/code/predict/1-confidence-vm5.py", line 43, in <module>
  net = RNN(filename = os.path.expanduser(NNET))
  File "SSSF/code/predict/RNN/RNN.py", line 147, in __init__
  )[0][1][::(1 if d == 0 else -1)] for d in range(Ndirs)], axis = 2)
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/scan_module/scan.py", line 1018, in scan
  local_op = scan_op.Scan(inner_inputs, new_outs, info)
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 214, in __init__
  [])
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line 1314, in cmodule_key_variables
  c_compiler)
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line 1364, in cmodule_key_
  numpy.core.multiarray._get_ndarray_c_version())
  AttributeError: 'module' object has no attribute '_get_ndarray_c_version'

• try switching to miniconda - noisemes-test.log:

  ImportError: No module named _multiarray_umath
  ImportError: No module named _multiarray_umath
  ImportError: No module named _multiarray_umath
  Traceback (most recent call last):
  File "SSSF/code/predict/1-confidence-vm5.py", line 21, in <module>
  from RNN import RNN
  File "SSSF/code/predict/RNN/RNN.py", line 2, in <module>
  import theano, theano.tensor as T
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/__init__.py", line 76, in <module>
  from theano.scan_module import scan, map, reduce, foldl, foldr, clone
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/scan_module/__init__.py", line 40, in <module>
  from theano.scan_module import scan_opt
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 59, in <module>
  from theano import tensor, scalar
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/tensor/__init__.py", line 7, in <module>
  from theano.tensor.subtensor import *
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/tensor/subtensor.py", line 27, in <module>
  import theano.gof.cutils  # needed to import cutils_ext
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/gof/cutils.py", line 319, in <module>
  compile_cutils()
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/gof/cutils.py", line 284, in compile_cutils
  preargs=args)
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", line 2213, in compile_str
  return dlimport(lib_filename)
  File "/home/vagrant/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", line 299, in dlimport
  rval = __import__(module_name, {}, {}, [module_name])
  ImportError: numpy.core.multiarray failed to import

  • Try switching back to (reinstalling) Anaconda: Now it works! Was this just a fluke? A bad install?

So, something is funky with versions of Python packages related to Theano and numpy

[riebling]

To verify the steps I took to 'fix' this, I tried from a fresh 'vagrant up', the test passed as above. Then these steps seemed to 'fix' the issue:

  155  rm -rf ~/anaconda
  156  ls
  157  wget -q https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.3.0-Linux-\
x86_64.sh
  158  bash Anaconda-2.3.0-Linux-x86_64.sh -b
  159  conda install numpy scipy mkl dill tabulate joblib cython=0.22.1 sphinx
  160  launcher/test.sh

What is INTERESTING to observe: although bootstrap.sh installs explicitly theano=0.8.2, after my brute force reinstall, the output of pip freeze | grep -i theano (is this even the right way to check Python package versions?!) shows a different Theano version:

numpy==1.10.2
vagrant@vagrant-ubuntu-trusty-64:~$ pip freeze | grep -i theano
You are using pip version 9.0.1, however version 19.0.1 is available.                                                  
You should consider upgrading via the 'pip install --upgrade pip' command.
Theano==0.7.0

I'm worried about hard coding / forcing Theano version just so Noisemes passes it's test, if it results in degraded/undesirable performance of other tools that use it: where did the hard dependency of theano=0.8.2 come from? (probably Yun)

[riebling]

Sure enough, now Diartk and VCM fail their selftests with Theano==0.7.0. Not the right solution :( I defer to people who are more Python users/experts to help figure out: is it OK the way we're mixing the use of pip install and conda install? Need to better understand this. It almost seems like bootstrap.sh may have swapped some of these.

Whoa, installing just theano=0.8.2 with conda, now Noismes, Diartk, Yunitator, VCM, WCE - ALL fail. Definitely not a 'fix'!

[riebling]

Ugh. This was for some reason running the major.reorganization branch. I swear I checked out main, I mean head, I mean master, when I started things. Hope that wasn't a problem as well.

[MarvinLvn]

I've seen that the major_reorganization branch had 3 commits ahead master (here) Could you integrate these modifications to the master branch ? That way I'll delete this old and fragile branch.

[riebling]

Ok, done! I'm not sure how my working directory ended up on major_reorganization. Good to be in sync again.