Cython-conditional setup.py leads to compilation errors

[yarden]

I just tried your new setup.py and it does not compile for me:

$ pip install .
Unpacking /Users/yarden/my_projects/pyhsmm
  Running setup.py (path:/var/folders/fk/qdjdmlm50pz_jt5lng_rkn480000gn/T/pip-sfcCj0-build/setup.py) egg_info for package from file:///Users/yarden/my_projects/pyhsmm

Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20.1 in /Users/yarden/anaconda/lib/python2.7/site-packages (from pyhsmm==0.1.3)
Requirement already satisfied (use --upgrade to upgrade): numpy in /Users/yarden/anaconda/lib/python2.7/site-packages (from pyhsmm==0.1.3)
Requirement already satisfied (use --upgrade to upgrade): scipy in /Users/yarden/anaconda/lib/python2.7/site-packages (from pyhsmm==0.1.3)
Requirement already satisfied (use --upgrade to upgrade): matplotlib in /Users/yarden/anaconda/lib/python2.7/site-packages (from pyhsmm==0.1.3)
Requirement already satisfied (use --upgrade to upgrade): nose in /Users/yarden/anaconda/lib/python2.7/site-packages (from pyhsmm==0.1.3)
Requirement already satisfied (use --upgrade to upgrade): pybasicbayes in /Users/yarden/anaconda/lib/python2.7/site-packages (from pyhsmm==0.1.3)
Installing collected packages: pyhsmm
  Running setup.py install for pyhsmm
    building 'pyhsmm/internals.hmm_messages_interface' extension
    gcc -fno-strict-aliasing -I/Users/yarden/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ipyhsmm/deps/Eigen3 -I/Users/yarden/anaconda/lib/python2.7/site-packages/numpy/core/include -I/Users/yarden/anaconda/include/python2.7 -c pyhsmm/internals/hmm_messages_interface.cpp -o build/temp.macosx-10.5-x86_64-2.7/pyhsmm/internals/hmm_messages_interface.o -O3 -std=c++11 -DNDEBUG -w -DHMM_TEMPS_ON_HEAP
    clang: error: no such file or directory: 'pyhsmm/internals/hmm_messages_interface.cpp'
    clang: error: no input files
    error: command 'gcc' failed with exit status 1
    Complete output from command /Users/yarden/anaconda/bin/python -c "import setuptools, tokenize;__file__='/var/folders/fk/qdjdmlm50pz_jt5lng_rkn480000gn/T/pip-sfcCj0-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/fk/qdjdmlm50pz_jt5lng_rkn480000gn/T/pip-vP3zDY-record/install-record.txt --single-version-externally-managed --compile:
    running install

running build

running build_py

creating build

creating build/lib.macosx-10.5-x86_64-2.7

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm

copying pyhsmm/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm

copying pyhsmm/models.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm

copying pyhsmm/parallel.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm/basic

copying pyhsmm/basic/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/basic

copying pyhsmm/basic/abstractions.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/basic

copying pyhsmm/basic/distributions.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/basic

copying pyhsmm/basic/models.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/basic

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

copying pyhsmm/internals/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

copying pyhsmm/internals/hmm_states.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

copying pyhsmm/internals/hsmm_inb_states.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

copying pyhsmm/internals/hsmm_states.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

copying pyhsmm/internals/initial_state.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

copying pyhsmm/internals/transitions.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/internals

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/animation.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/concentration-resampling.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hmm-EM.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hmm-separatetrans.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hmm.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hsmm-geo.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hsmm-possiblechangepoints-meanfield.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hsmm-possiblechangepoints-separatetrans.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hsmm-possiblechangepoints.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/hsmm.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/meanfield.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

copying pyhsmm/examples/svi.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm/plugins

copying pyhsmm/plugins/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/plugins

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm/testing

copying pyhsmm/testing/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/testing

copying pyhsmm/testing/test_hmm_geweke.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/testing

copying pyhsmm/testing/test_hmm_likelihood.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/testing

creating build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/general.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/plot.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/profiling.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/setup.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/stats.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/testing.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/util/text.py -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/util

copying pyhsmm/examples/example-data.txt -> build/lib.macosx-10.5-x86_64-2.7/pyhsmm/examples

running build_ext

building 'pyhsmm/internals.hmm_messages_interface' extension

creating build/temp.macosx-10.5-x86_64-2.7

creating build/temp.macosx-10.5-x86_64-2.7/pyhsmm

creating build/temp.macosx-10.5-x86_64-2.7/pyhsmm/internals

gcc -fno-strict-aliasing -I/Users/yarden/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ipyhsmm/deps/Eigen3 -I/Users/yarden/anaconda/lib/python2.7/site-packages/numpy/core/include -I/Users/yarden/anaconda/include/python2.7 -c pyhsmm/internals/hmm_messages_interface.cpp -o build/temp.macosx-10.5-x86_64-2.7/pyhsmm/internals/hmm_messages_interface.o -O3 -std=c++11 -DNDEBUG -w -DHMM_TEMPS_ON_HEAP

clang: error: no such file or directory: 'pyhsmm/internals/hmm_messages_interface.cpp'

clang: error: no input files

error: command 'gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /Users/yarden/anaconda/bin/python -c "import setuptools, tokenize;__file__='/var/folders/fk/qdjdmlm50pz_jt5lng_rkn480000gn/T/pip-sfcCj0-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/fk/qdjdmlm50pz_jt5lng_rkn480000gn/T/pip-vP3zDY-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /var/folders/fk/qdjdmlm50pz_jt5lng_rkn480000gn/T/pip-sfcCj0-build
Storing debug log for failure in /Users/yarden/.pip/pip.log

I can try to fix it while integrating travis. Relying on custom flags like --with-cython I think is a bad idea, because package mangers won't know about these. It also seems like the flag is unnecessary. python setup.py build_ext or python setup.py sdist can figure out if Cython needs to be called and if so run it.

[mattjj]

My thinking is that --with-cython can be for building from the git repository with python setup.py and not pip, while the distribution on pypi won't need it because it will have the .cpp files included. I don't think supporting pip install . is absolutely necessary if we have those two clear paths working.

However, you're right that a fully automatic version would be a better solution. So should we just call cython if the .cpp files aren't present? How would developers signal that they want to force cython to be called (even if the .cpp files are present)?

[yarden]

The latest at https://github.com/mattjj/pyhsmm/pull/39 should now do this. It doesn't rely on cython if it's not present (i.e. if you're in a source distribution), it error checks to see that the .cpp files are actually there, and it works with pip install . so it can support automation.

[mattjj]

Moving over to #39 (when I get a chance to respond!).