Nosetests Failure

[davidgwyrick]

Hey Scott,

I'm getting 4 errors on the nosetests after doing a fresh install of ssm in a new conda environment. From the looks of it, they all seem to be related to an import error of 'hmm_expected_states'. Is this related to cython or something, since messages.py is cythonized? I followed your instructions on the install and tried a few variations but to no avail. Any ideas?

Cheers, David

[bantin]

Hi @davidgwyrick , my name is Ben -- I'm working on SSM with Scott. Thanks for reporting this, someone else actually mentioned this to me before, but I wasn't able to reproduce it on my own machine. It seems like it was caused by switching messages.py to use Numba, rather than Cython, but I don't know exactly why that is causing this. This is a good reminder to make sure we get this fixed.

Could you try this and let me know the result: create a new clean virtualenv with pip (not a conda environment) which doesn't have SSM installed, and try installing there?

The steps should look something like this: cd ~ pip install virtualenv virtualenv test_env --no-site-packages source ~/test_env/bin/activate pip install numpy cython cd /directory/where/ssm/is pip install -e .

Let me know if that resolves the error.

[davidgwyrick]

Hey Ben, thanks for getting back to me. Now it looks like there are other syntax errors that cause the nosetests to fail.

`(test_env) dwyrick:ssm$ nosetests EEEEE

ERROR: Failure: SyntaxError (invalid syntax (hmm.py, line 499))

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/dwyrick/Git/ssm/ssm/init.py", line 3, in from .hmm import File "/home/dwyrick/Git/ssm/ssm/hmm.py", line 499 def init(self, K, D, , M=0, init_state_distn=None, ^ SyntaxError: invalid syntax

====================================================================== ERROR: Failure: ImportError (No module named autograd.numpy)

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/dwyrick/Git/ssm/tests/test_basics.py", line 3, in import autograd.numpy as np ImportError: No module named autograd.numpy

====================================================================== ERROR: Failure: ImportError (No module named autograd.numpy)

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/dwyrick/Git/ssm/tests/test_hmm_gradients.py", line 1, in import autograd.numpy as np ImportError: No module named autograd.numpy

====================================================================== ERROR: Failure: SyntaxError (invalid syntax (hmm.py, line 499))

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/dwyrick/Git/ssm/tests/test_lds.py", line 2, in import ssm File "/home/dwyrick/Git/ssm/ssm/init.py", line 3, in from .hmm import File "/home/dwyrick/Git/ssm/ssm/hmm.py", line 499 def init(self, K, D, , M=0, init_state_distn=None, ^ SyntaxError: invalid syntax

====================================================================== ERROR: Failure: SyntaxError (only named arguments may follow *expression (test_stats.py, line 48))

Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/dwyrick/Git/ssm/tests/test_stats.py", line 48 x = npr.randn(shp, D) SyntaxError: only named arguments may follow expression

---

Ran 5 tests in 0.010s

FAILED (errors=5) (test_env) dwyrick:ssm$`

[bantin]

Hi David -- I think these new errors are probably a python 2.X vs. python 3.X issue. In the output, I see some /usr/lib/python2.7/ which I think means you are running Python 2.7 (although it might be possible that you're running Python 3 but somehow the old packages are being called). Since there are some syntax differences between Python 2 and 3, this might explain some of what you're seeing.

We only support and test with Python 3 (although we certainly should be more explicit about this in the documentation.)

Can you check if you're using Python 3 (preferably 3.7) and let me know if that fixes things?

[davidgwyrick]

Weird, I had created the virtual environment with the python3.7 option, but I guess nosetests was still pointing to the python2.7 package. I don't usually use pip for my package manager, but I think I got things to work..

although when I run the nosetests, i do get the following UserWarnings and the following new* error.

/home/dwyrick/Git/ssm/ssm/optimizers.py:132: UserWarning: L-BFGS-B failed with message:
b'STOP: TOTAL NO. of ITERATIONS REACHED LIMIT'
  warn("{} failed with message:\n{}".format(method, result.message))

/home/dwyrick/Git/ssm/ssm/messages.py:190: RuntimeWarning: divide by zero encountered in log
  expected_joints = alphas[:-1,:,None] + betas[1:,None,:] + ll[1:,None,:] + np.log(Ps)

`====================================================================== ERROR: test_lds.test_lds_sample_and_fit

Traceback (most recent call last): File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/home/dwyrick/Git/ssm/tests/test_lds.py", line 486, in test_lds_sample_and_fit num_iters=2) File "/home/dwyrick/Git/ssm/ssm/util.py", line 110, in wrapper return f(self, datas, inputs=inputs, masks=masks, tags=tags, kwargs) File "/home/dwyrick/Git/ssm/ssm/lds.py", line 856, in fit self.initialize(datas, inputs, masks, tags, num_iters=num_init_iters) File "/home/dwyrick/Git/ssm/ssm/util.py", line 110, in wrapper return f(self, datas, inputs=inputs, masks=masks, tags=tags, kwargs) File "/home/dwyrick/Git/ssm/ssm/lds.py", line 184, in initialize method="em", num_iters=num_iters) File "/home/dwyrick/Git/ssm/ssm/util.py", line 110, in wrapper return f(self, datas, inputs=inputs, masks=masks, tags=tags, kwargs) File "/home/dwyrick/Git/ssm/ssm/hmm.py", line 486, in fit return _fitting_methods[method](datas, inputs=inputs, masks=masks, tags=tags, kwargs) File "/home/dwyrick/Git/ssm/ssm/hmm.py", line 449, in _fit_em self.transitions.m_step(expectations, datas, inputs, masks, tags, transitions_mstep_kwargs) File "/home/dwyrick/Git/ssm/ssm/transitions.py", line 297, in m_step Transitions.m_step(self, expectations, datas, inputs, masks, tags, kwargs) File "/home/dwyrick/Git/ssm/ssm/transitions.py", line 71, in m_step state=optimizer_state, full_output=True, kwargs) File "/home/dwyrick/Git/ssm/ssm/optimizers.py", line 127, in _generic_minimize kwargs) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/scipy/optimize/_minimize.py", line 610, in minimize callback=callback, options) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py", line 345, in _minimize_lbfgsb f, g = func_and_grad(x) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py", line 296, in func_and_grad g = jac(x, args) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/autograd/wrap_util.py", line 20, in nary_f return unary_operator(unary_f, x, nary_op_args, nary_op_kwargs) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/autograd/differential_operators.py", line 29, in grad return vjp(vspace(ans).ones()) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/autograd/core.py", line 14, in vjp def vjp(g): return backward_pass(g, end_node) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/autograd/core.py", line 21, in backward_pass ingrads = node.vjp(outgrad[0]) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/autograd/core.py", line 67, in return lambda g: (vjp(g),) File "/home/dwyrick/Git/ssm/test_env/lib/python3.7/site-packages/autograd/scipy/stats/dirichlet.py", line 13, in g * (alpha - 1) / x, FloatingPointError: invalid value encountered in true_divide -------------------- >> begin captured stdout << --------------------- `

[bantin]

Okay great, this looks like progress. The user warnings are nothing to worry about -- they come up because we sometimes have to take the log of zero. Hopefully we'll be able to squash them to clean things up in a future commit.

I haven't seen the divide by zero issue happen in dirichlet.py. I'll have to see if I can reproduce that. Just to double check, this is on the latest branch of master?

[davidgwyrick]

Indeed, this is the latest branch of master. It also looks like the notebook examples are running fine.