import error for pyhsmm examples

[dchouren]

Hey Matt, thanks for your work here.

I'm trying to run some of the examples and I'm getting ImportError: No module named hmm_messages_interface.

Full trace is here:

---

python hmm.py
/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/transitions.py:16: UserWarning: using slow transition counting
  warn('using slow transition counting')

This demo shows how HDP-HMMs can fail when the underlying data has state
persistence without some kind of temporal regularization (in the form of a
sticky bias or duration modeling): without setting the number of states to be
the correct number a priori, lots of extra states can be intsantiated.

BUT the effect is much more relevant on real data (when the data doesn't exactly
fit the model). Maybe this demo should use multinomial emissions...

Traceback (most recent call last):
  File "hmm.py", line 64, in <module>
    posteriormodel.resample_model()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/models.py", line 431, in resample_model
    self.resample_states(num_procs=num_procs)
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/models.py", line 456, in resample_states
    s.resample()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 356, in resample
    return self.resample_normalized()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 352, in resample_normalized
    alphan = self.messages_forwards_normalized()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 342, in messages_forwards_normalized
    self._messages_forwards_normalized(self.trans_matrix,self.pi_0,self.aBl)
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 605, in _messages_forwards_normalized
    from hmm_messages_interface import messages_forwards_normalized
ImportError: No module named hmm_messages_interface

---

I think this must be related to an issue I had running python setup.py install. While I am able to import pyhsmm, the install command had an error in it:

building 'pyhsmm.internals.hmm_messages_interface' extension
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -Ideps/ -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/usr/local/lib/python2.7/site-packages/numpy/core/include -c pyhsmm/internals/hmm_messages_interface.cpp -o build/temp.macosx-10.10-intel-2.7/pyhsmm/internals/hmm_messages_interface.o -std=c++11 -O3 -w -DNDEBUG -DHMM_TEMPS_ON_HEAP
pyhsmm/internals/hmm_messages_interface.cpp:5484:672: error: cannot initialize a
      parameter of type 'int *' with an rvalue of type
      '__pyx_t_5numpy_int32_t *' (aka 'long *')
  ...(&(*__Pyx_BufPtrCContig1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_stateseq.rcbuffer->pybuffer.buf, __pyx_t_20, __pyx_pybuffernd_stateseq.diminfo[0].strides)))...
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pyhsmm/internals/hmm_messages.h:273:22: note: passing argument to parameter
      'stateseq' here
            IntType *stateseq, FloatType *randseq)
                     ^
pyhsmm/internals/hmm_messages_interface.cpp:5875:680: error: cannot initialize a
      parameter of type 'int *' with an rvalue of type
      '__pyx_t_5numpy_int32_t *' (aka 'long *')
  ...(&(*__Pyx_BufPtrCContig1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_stateseq.rcbuffer->pybuffer.buf, __pyx_t_20, __pyx_pybuffernd_stateseq.diminfo[0].strides)))...
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pyhsmm/internals/hmm_messages.h:273:22: note: passing argument to parameter
      'stateseq' here
            IntType *stateseq, FloatType *randseq)
                     ^
pyhsmm/internals/hmm_messages_interface.cpp:9632:452: error: cannot initialize a
      parameter of type 'int *' with an rvalue of type
      '__pyx_t_5numpy_int32_t *' (aka 'long *')
  ...(&(*__Pyx_BufPtrCContig1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_stateseq.rcbuffer->pybuffer.buf, __pyx_t_17, __pyx_pybuffernd_stateseq.diminfo[0].strides)))...
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pyhsmm/internals/hmm_messages.h:283:22: note: passing argument to parameter
      'stateseq' here
            IntType *stateseq, FloatType *randseq)
                     ^
pyhsmm/internals/hmm_messages_interface.cpp:9988:456: error: cannot initialize a
      parameter of type 'int *' with an rvalue of type
      '__pyx_t_5numpy_int32_t *' (aka 'long *')
  ...(&(*__Pyx_BufPtrCContig1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_stateseq.rcbuffer->pybuffer.buf, __pyx_t_17, __pyx_pybuffernd_stateseq.diminfo[0].strides)))...
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pyhsmm/internals/hmm_messages.h:283:22: note: passing argument to parameter
      'stateseq' here
            IntType *stateseq, FloatType *randseq)
                     ^
pyhsmm/internals/hmm_messages_interface.cpp:10818:442: error: cannot initialize
      a parameter of type 'int *' with an rvalue of type '__pyx_t_5numpy_int32_t
      *' (aka 'long *')
  ...(&(*__Pyx_BufPtrCContig1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_stateseq.rcbuffer->pybuffer.buf, __pyx_t_6, __pyx_pybuffernd_stateseq.diminfo[0].strides))));
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pyhsmm/internals/hmm_messages.h:288:22: note: passing argument to parameter
      'stateseq' here
            IntType *stateseq)
                     ^
pyhsmm/internals/hmm_messages_interface.cpp:11014:448: error: cannot initialize
      a parameter of type 'int *' with an rvalue of type '__pyx_t_5numpy_int32_t
      *' (aka 'long *')
  ...(&(*__Pyx_BufPtrCContig1d(__pyx_t_5numpy_int32_t *, __pyx_pybuffernd_stateseq.rcbuffer->pybuffer.buf, __pyx_t_6, __pyx_pybuffernd_stateseq.diminfo[0].strides))));
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pyhsmm/internals/hmm_messages.h:288:22: note: passing argument to parameter
      'stateseq' here
            IntType *stateseq)
                     ^
6 errors generated.
setup.py:35: UserWarning: Failed to build extension modules
  warn('Failed to build extension modules')

For gcc I have

gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.0 (clang-700.0.72)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

Is there something else I need to do with python setup.py install? Thanks!

[dchouren]

Whoo, fixed! The main problem seems to have been trying to set it up in a virtualenv.

I did some more digging into other issues and #22 seemed similar. I followed your suggestion there to try these commands

git clone --recursive https://github.com/mattjj/pyhsmm.git
cd pyhsmm
python setup.py build_ext --inplace
find . -iname '*.so'

For anyone reading this, the commands only worked properly for me outside my virtualenv. I had to clone pybasicbayes again inside pyhsmm, but the examples seem to work now. No virtualenv is slightly annoying, but I'll take it.

[dchouren]

Actually, I just remembered I will have to use my virtualenv later for nonlocal development.

So I guess the question now is why activating a virtualenv causes the following error trace when running hmm.py (or any example):

Traceback (most recent call last):
  File "examples/hmm.py", line 51, in <module>
    posteriormodel.resample_model()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/models.py", line 431, in resample_model
    self.resample_states(num_procs=num_procs)
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/models.py", line 456, in resample_states
    s.resample()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 356, in resample
    return self.resample_normalized()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 352, in resample_normalized
    alphan = self.messages_forwards_normalized()
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 342, in messages_forwards_normalized
    self._messages_forwards_normalized(self.trans_matrix,self.pi_0,self.aBl)
  File "/Users/daway/Documents/Princeton/Thesis/seldon/lib/python2.7/site-packages/pyhsmm-0.1.5-py2.7-macosx-10.10-intel.egg/pyhsmm/internals/hmm_states.py", line 605, in _messages_forwards_normalized
    from hmm_messages_interface import messages_forwards_normalized
ImportError: No module named hmm_messages_interface

[mattjj]

Glad you like the library! Hope we can get it to work for you. Compilation problems are a pain.

You shouldn't need to clone pybasicbayes inside pyhsmm (though I used to organize things that way). It should be importable (e.g. from doing pip install pybasicbayes or cloning it and doing python setup.py install) but that should be the only requirement.

The main problem looks like a compilation issue. The module hmm_messages_interface gets created during the compilation process, and a compilation error would prevent it from being created. (I wrote the setup.py file to continue installing even if the extension modules failed to build because you can still use parts of the library without the low-level code, though the examples won't work and the library will be less useful.)

The error seems to be that your system doesn't like confusing an int * with a long *, and it sounds like this problem from Issue 23. In that thread, it sounds like that particular compilation issue was solved by using gcc 4.8, but there are other problems so it's not so clear to me what happened. You may want to read the rest of that thread if you haven't already.

The problem is either in how the C is getting generated via cython or in some difference between your compilation toolchain and mine. If it's a problem with the cython code generation, the first step would probably be to ensure you're on the most up-to-date cython. You can check with

python -c 'import cython; print cython.__version__'

I'm on 0.23.4. If you're on a much older version, it would be good to download and install anaconda or miniconda and installing cython through conda. Even if you're on that same version of cython, it may be a good idea to try using a fresh conda-installed version, since your error looks like some disagreement about the details of your platform between cython and your compiler.

If it's a problem with compiling the generated code, a first step would be to use the same compiler as I'm using (as seemed to help in #23). I have tested clang at some point before, and yours looks like a recent version, but I always use gcc. (On Mac OS, the name gcc is symlinked to a clang version, which is why it prints out that Apple LLVM stuff.) So a first step could be to install a recent gcc version (4.8 or later is required) through homebrew or macports and compiling with that (either by changing the gcc symlink or setting the CC environment variable before you run the setup.py build script).

If those two steps (updating cython and switching to gcc) don't help, I can try to reproduce things on my system.

[dchouren]

Awesome, it wasn't working with gcc 4.9 but I downgraded to 4.8 and it seems to be good now. Having an issue with plots (matplotlib backends) but that's an unrelated issue I can figure out on my own.

Thanks!

[mattjj]

Hrm I'd really like it to work with 4.9. Thanks for letting me know. If you find anything else related, it'd be great to update this thread.

Glad to hear it looks like it's working!

[mattjj]

I set up travis to test gcc 4.9 on Ubuntu (in addition to 4.8) and the build worked, so there must be something different about either Mac OS or your particular setup. I'll update this thread if I get a chance to test on Mac OS, but at least things seem to work as intended on linux.

(On my main work machine I'm using gcc 5.3.0, which also works.)

[dchouren]

I did a little more digging around and it seems to work only on a remote linux server and not on a local Mac environment. Reopening this in hopes of getting the local setup working as well.

The remote server has gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC). Local setup has gcc version 4.8.5 (MacPorts gcc48 4.8.5_0). I linked gcc locally with 'sudo port select --set gcc mp-gcc45', but I'm a bit of a newb at setting things up, so let me know if I'm not actually using the right gcc here.

Both setups have cython 0.23.4. Local setup has Anaconda, not sure what the linux server is using.

The full error trace for build_ext is

python setup.py build_ext
running build_ext
building 'pyhsmm.internals.hmm_messages_interface' extension
gcc-4.8 -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -Ideps/ -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/usr/local/lib/python2.7/site-packages/numpy/core/include -c pyhsmm/internals/hmm_messages_interface.cpp -o build/temp.macosx-10.10-intel-2.7/pyhsmm/internals/hmm_messages_interface.o -std=c++11 -O3 -w -DNDEBUG -DHMM_TEMPS_ON_HEAP
gcc-4.8: error: unrecognized command line option '-Wshorten-64-to-32'
setup.py:35: UserWarning: Failed to build extension modules
  warn('Failed to build extension modules')

Not sure what gcc-4.8: error: unrecognized command line option '-Wshorten-64-to-32' is... have you seen that before?

[mattjj]

I have seen that one before. It's a clang-specific flag, meaning the python you're using was built with clang in mind. I believe the flags get generated based on how Python itself was compiled.

Are you sure you're using the anaconda python and not the system python? You can double check with which python and maybe ls -l $(which python) to check for symlinks.

[mattjj]

See e.g. this issue.

[dchouren]

hm.. so I was using a virtualenv so you were right, I was using a system python. But when I deactivated that and checked I was using Anaconda, now I get a different trace:

> python setup.py build_ext
running build_ext
building 'pyhsmm.internals.hmm_messages_interface' extension
creating build/temp.macosx-10.5-x86_64-2.7
creating build/temp.macosx-10.5-x86_64-2.7/pyhsmm
creating build/temp.macosx-10.5-x86_64-2.7/pyhsmm/internals
gcc-4.8 -fno-strict-aliasing -I/Users/daway/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ideps/ -I/Users/daway/anaconda/include/python2.7 -I/usr/local/lib/python2.7/site-packages/numpy/core/include -c pyhsmm/internals/hmm_messages_interface.cpp -o build/temp.macosx-10.5-x86_64-2.7/pyhsmm/internals/hmm_messages_interface.o -std=c++11 -O3 -w -DNDEBUG -DHMM_TEMPS_ON_HEAP
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//ccsep4oZ.s:110223:expecting string instruction after `rep'
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//ccsep4oZ.s:110248:expecting string instruction after `rep'
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//ccsep4oZ.s:110298:expecting string instruction after `rep'
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//ccsep4oZ.s:110635:expecting string instruction after `rep'
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//ccsep4oZ.s:110660:expecting string instruction after `rep'
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//ccsep4oZ.s:110710:expecting string instruction after `rep'
setup.py:35: UserWarning: Failed to build extension modules
  warn('Failed to build extension modules')

[mattjj]

Maybe try a clean first (remove the build directory, all .so files, etc).

[mattjj]

You should be able to run python setup.py clean and/or python setup.py clean --all, but I haven't tested that much.

[dchouren]

Still running into the same problem :(

[dchouren]

Tried switching to gcc-4.9 and get this

running build_ext
building 'pyhsmm.internals.hmm_messages_interface' extension
gcc-4.9 -fno-strict-aliasing -I/Users/daway/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ideps/ -I/Users/daway/anaconda/include/python2.7 -I/usr/local/lib/python2.7/site-packages/numpy/core/include -c pyhsmm/internals/hmm_messages_interface.cpp -o build/temp.macosx-10.5-x86_64-2.7/pyhsmm/internals/hmm_messages_interface.o -std=c++11 -O3 -w -DNDEBUG -DHMM_TEMPS_ON_HEAP
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/var/folders/1c/vwbjnwn938gd9lgrqkdfj5300000gn/T//cclM3HKd.s:20670:suffix or operands invalid for `movq'
setup.py:35: UserWarning: Failed to build extension modules
  warn('Failed to build extension modules')

[mattjj]

I think this try/except in setup.py is suppressing useful error information here. Can you comment out or delete that whole run method and see if something more useful gets printed?

[mattjj]

I'm going to try printing the exception traceback even when proceeding with the installation process using these ideas.

[dchouren]

commented that block out but get same error trace

[dchouren]

Actually, I lied, but the only difference is instead of

setup.py:35: UserWarning: Failed to build extension modules warn('Failed to build extension modules')

I get this

error: command 'gcc-4.8' failed with exit status 1

[mattjj]

There are a lot of google hits for that error message, "suffix or operands invalid for `movq". I haven't seen this before, so those are probably your best bet.

[dchouren]

This has to be some awful joke... I just reran python setup.py install except with sudo and it worked.

haha, what a fix