Setup tools not properly installing

[tansey]

Hi,

I was failing to get this to install/run on both os x 10.9 and ubuntu with the recommended:

cd pyhsmm
python setup.py build_ext --inplace

Followed by either python setup.py install or cp -r pyhsmm /usr/local/lib/python2.7/site-packages/.

So I decided to try and fix the repo to make it properly install.

Here is the ordered list of what I've done thus far (from inside the cloned pyhsmm folder, on a relatively fresh install of Mavericks, with numpy/scipy/matplotlib already setup):

change setup.py:
pyhsmm.util.cyutil
setup(
    name='pyhsmm',
    url='https://github.com/mattjj/pyhsmm',
    packages=['pyhsmm', 'pyhsmm.basic', 'pyhsmm.basic.pybasicbayes', 'pyhsmm.basic.pybasicbayes.util', 'pyhsmm.basic.pybasicbayes.internals', 'pyhsmm.plugins', 'pyhsmm.util', 'pyhsmm.internals', 'pyhsmm.testing'],
    ext_modules=ext_modules,
    include_dirs=[np.get_include(),],
)

mkdir pyhsmm
mv __init__.py pyhsmm/
mv basic/ pyhsmm/
mv deps/ pyhsmm/
mv examples/ pyhsmm/
mv internals/ pyhsmm/
mv models.py pyhsmm/
mv parallel.py pyhsmm/
mv testing/ pyhsmm/
mv util/ pyhsmm/
cd pyhsmm/deps
mkdir eigen_build
cd eigen_build
homebrew install cmake
cmake ../Eigen3/
sudo make install
cd ../../../

change hmm_messages.h, hsmm_messages.h, nptypes.h ==> include <eigen3/Eigen/Core>
prepend pyhsmm. to pyhsmm/models.py (including line 267)
prepend pyhsmm.basic to pyhsmm/basic/models.py
prepend pyhsmm.basic.pybasicbayes. to all imports from files in pyhsmm/basic/pybasicbayes/
prepend pyhsmm. to all imports in pyhsmm/__init__.py
prepend pyhsmm.internals. to line 1044 of pyhsmm/internals/states.py

python setup.py build_ext --inplace
python setup.py install

cd pyhsmm/examples
python hsmm.py

The result is an error:

Traceback (most recent call last):
  File "hsmm.py", line 54, in <module>
    posteriormodel.resample_model()
  File "/usr/local/lib/python2.7/site-packages/pyhsmm/models.py", line 200, in resample_model
    self.resample_states()
  File "/usr/local/lib/python2.7/site-packages/pyhsmm/models.py", line 222, in resample_states
    s.resample()
  File "/usr/local/lib/python2.7/site-packages/pyhsmm/internals/states.py", line 840, in resample
    betal, betastarl = self.messages_backwards()
  File "/usr/local/lib/python2.7/site-packages/pyhsmm/internals/states.py", line 1044, in messages_backwards
    from pyhsmm.internals.hsmm_messages_interface import messages_backwards_log
ImportError: No module named hsmm_messages_interface

Unlike the other errors I've gotten along the way, I don't know how to fix this one. All there is in internals is hsmm_messages_interface.cpp and hsmm_messages_interface.pyx -- and it seems like I'm unable to get pyximport to help me out with finding the interface anywhere.

Any ideas how to get the import to work? Or is there a different, known way to install the package on OS X?

[mattjj]

As you noticed, the current setup.py doesn't provide any of the usual packaging information. I usually just symlink the repository into my PYTHONPATH where I keep personal code. I'm up for making it properly installable at some point, but since that requires some tweaks to the directory structure, for the purposes of debugging this problem it may be easiest to focus on building pyhsmm in place and not using distutils to install it anywhere yet. (I should also mention this is code I use in my research and not a nicely packaged library.)

It looks like that ImportError is coming from hsmm_messages_interface.pyx not being built. It should be built by the python setup.py build_ext --inplace command, specifically by the recursive globbing in this line of the current setup.py:

ext_modules = cythonize('**/*.pyx')

I'm not sure I'm tracking your directory reorganization, but these commands work for me on a fresh machine:

git clone --recursive https://github.com/mattjj/pyhsmm.git
cd pyhsmm
python setup.py build_ext --inplace
find . -iname '*.so'

The last line prints out this:

./basic/pybasicbayes/util/cstats.so
./internals/hmm_messages_interface.so
./internals/hsmm_messages_interface.so
./util/cstats.so

Can you try running those commands in a fresh directory and either tell me the error or verify that the .so files get built?

One preliminary guess, since you didn't mention which compiler you're using and you're on OS X, is that you're building with an older clang. You might also try to replace the build line with python setup.py build_ext --inplace --with-old-clang. However, I usually build with g++ (and I find it generates better code), so if you have g++-4.7 or later you should probably build with something like CC=$(which g++) python setup.py build_ext --inplace (assuming you're using bash or zsh).

For the record, I'm on OS X 10.8, and I just tested that the --with-old-clang works with my /usr/bin/clang, which reports

Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix

[mattjj]

Once we figure out how to get the .so files built properly, you can try importing pyhsmm from the directory in which you cloned it. If that works, then you should be able to cp -r the whole repository into your site-packages (I just checked), though I usually just set my PYTHONPATH to include /Users/mattjj/Library/Python and do a ln -s pyhsmm /Users/mattjj/Library/Python/pyhsmm.

[tansey]

Thanks Matt.

I'm on OS X 10.9, building with g++-4.9 (installed via Homebrew). I get the same output as you after running the initial build:

./basic/pybasicbayes/util/cstats.so
./internals/hmm_messages_interface.so
./internals/hsmm_messages_interface.so
./util/cstats.so

If I then try to import pyhsmm from the directory in which I cloned the project, I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named hsmm

If I copy it to my site-packages:

sudo cp -r pyhsmm/ /usr/local/lib/python2.7/site-packages/

It successfully imports. However, if I then navigate to pyhsmm/examples/ and try running python hsmm.py, I again get:

> python hsmm.py 
Traceback (most recent call last):
  File "hsmm.py", line 7, in <module>
    import pyhsmm
ImportError: No module named pyhsmm

I'm not sure if there's an easier way to run a full test to make sure everything is properly installed, but this is the path I went down originally and when I got here I started digging into the setuptools configuration.

[mattjj]

You're trying to import a module named pyhsmm, not hsmm, right? Just to check, I mean this kind of thing from the directory where you ran the clone command (one level up from where you ran the find command):

$ ls
pyhsmm/
$ python
>>> import pyhsmm

Does that give you an ImportError about "hsmm"?

[tansey]

Right that's what I'm doing.

Also, I just tried adding the top-level directory to my PYTHONPATH and now it works from any directory. What's strange is that if I cp -r pyhsmm /to/anywhere then set my PYTHONPATH to /to/anywhere, it no longer works.

[mattjj]

Any idea why the ImportError says "No module named hsmm"?

What does this command print?

ls /usr/local/lib/python2.7/site-packages/pyhsmm

[tansey]

Not sure. I just walked through recreating and copying it over to site-packages again and now it works, so maybe I made a mistake somewhere along the way the first time.

Either way, it works now! I'll go ahead and close the issue. Thanks Matt!

[mattjj]

Great, I hope it keeps working! Lacking proper python packaging makes things harder than they should be though, and has probably scared less adventurous people away. I'll bump that up on my long to-do list, though pull requests are also welcome :)