search

machenslab / dPCA

An implementation of demixed Principal Component Analysis (a supervised linear dimensionality reduction technique)
MIT License
279 stars 94 forks source link

Fix for setup.py on OS X. #10

Closed cboulay closed 7 years ago

cboulay commented 7 years ago

Use distutils Extension instead of cythonize because include_dirs kwarg is ignored in setup() but is respected in Extension().

See thread here.

cboulay commented 7 years ago

Don't merge this PR yet. It's still not working properly. I can't import dPCA.

Building module dPCA.nan_shuffle failed: ["distutils.errors.CompileError: command 'clang' failed with exit status 1\n"]

But it was built during setup.py install...

cboulay commented 7 years ago

I have a simpler fix that is much closer to the original. I will force push it soon. But I'm still getting errors on from dpca import dPCA.

Here is the output of python3 setup.py install:

python3 setup.py install
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/dPCA
copying dPCA/__init__.py -> build/lib/dPCA
copying dPCA/dPCA.py -> build/lib/dPCA
running install_lib
creating /usr/local/lib/python3.6/site-packages/dPCA
copying build/lib/dPCA/__init__.py -> /usr/local/lib/python3.6/site-packages/dPCA
copying build/lib/dPCA/dPCA.py -> /usr/local/lib/python3.6/site-packages/dPCA
byte-compiling /usr/local/lib/python3.6/site-packages/dPCA/__init__.py to __init__.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/site-packages/dPCA/dPCA.py to dPCA.cpython-36.pyc
running install_egg_info
Writing /usr/local/lib/python3.6/site-packages/dPCA-0.1-py3.6.egg-info
running install
running build
running build_ext
building 'dPCA.nan_shuffle' extension
creating build/temp.macosx-10.12-x86_64-3.6
creating build/temp.macosx-10.12-x86_64-3.6/dPCA
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/lib/python3.6/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c dPCA/nan_shuffle.c -o build/temp.macosx-10.12-x86_64-3.6/dPCA/nan_shuffle.o
[snip nan_shuffle.c compile warnings]
45 warnings generated.
creating build/lib.macosx-10.12-x86_64-3.6
creating build/lib.macosx-10.12-x86_64-3.6/dPCA
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk build/temp.macosx-10.12-x86_64-3.6/dPCA/nan_shuffle.o -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/sqlite/lib -o build/lib.macosx-10.12-x86_64-3.6/dPCA/nan_shuffle.cpython-36m-darwin.so
running install_lib
copying build/lib.macosx-10.12-x86_64-3.6/dPCA/nan_shuffle.cpython-36m-darwin.so -> /usr/local/lib/python3.6/site-packages/dPCA
running install_egg_info
Writing /usr/local/lib/python3.6/site-packages/UNKNOWN-0.0.0-py3.6.egg-info
cboulay commented 7 years ago

OK I think I fixed the problem(s). First, the above problem about include_dirs. Second, cython wasn't generating the .c file because it was already in the repo. Unfortunately, this .c file was incorrect for my system. I removed it then added it to .gitignore. Now cython generates the .c file, which it then compiles, and the compiled output works correctly on my system so it can be imported properly.

I can now python3 setup.py install then in ipython run through the jupyter notebook example.

wielandbrendel commented 7 years ago

@cboulay thanks for the fix 👍! I am currently travelling, but I am taking a look beginning of next week.

wielandbrendel commented 7 years ago

@cboulay I tested your PR today and I am running into problems under Linux ('Installing collected packages: UNKNOWN'). Before we dig into any specifics: what was your original problem? I tested the installation on both Mac and Linux, and it worked fine on both. One crucial difference not documented is that I am using pip install instead of python setup.py install, which seems to be the recommended way to install packages these days. I'll update the docs.

cboulay commented 7 years ago

The problem was that setup was failing to compile the nan_shuffle module because it could not find the numpy headers. The include_dirs argument/dict key passed to setup() was not propagating properly. This is fixed by supplying include_dirs as an argument to the Extension directly, not to setup().

However, even after fixing this, I was still running into problems when trying to import the dPCA module because in pyximport.install(setup_args={'include_dirs': np.get_include()}), once again the headers could not be found.

I fixed this second problem by dumping the cython-generated .c file and regenerating it. I didn't think of it at the time, but it's likely this fix (dumping the .c file) would also fix the first problem.

I'll try pip3 install from a fresh clone. If that doesn't work then I'll try fixing only the .c file and see if that works.

wielandbrendel commented 7 years ago

I agree with removing the C-file (and the corresponding update to gitignore). Lets see if pip install solves your problem. Once everything works I'll push the dPCA package to pypi to simplify the installation.

cboulay commented 7 years ago

No, neither of those fixes worked. 

chadwickboulay$ pip3 install .
Processing /Volumes/STORE/Tools/Analysis/dPCA/python
Installing collected packages: dPCA
  Running setup.py install for dPCA ... error
[snip]
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c dPCA/nan_shuffle.c -o build/temp.macosx-10.12-x86_64-3.6/dPCA/nan_shuffle.o
    dPCA/nan_shuffle.c:444:10: fatal error: 'numpy/arrayobject.h' file not found
    #include "numpy/arrayobject.h"
             ^
    1 error generated.
    error: command 'clang' failed with exit status 1

Edit: Similar result with python3 setup.py install

wielandbrendel commented 7 years ago

Can you do "sudo pip3 install -v" and paste the output? Thanks! Which OS are you using? Same problem with pip2?

cboulay commented 7 years ago

I minimized my changes to setup.py and tried again. It still installs for me. Something worth noting: In this current version, nan_shuffle doesn't have any package information and gets installed in the root of site-packages as UNKNOWN-0.0.0-py3.6.egg-info. In my previous version, it was installed within dPCA because it was set as an external module in the dPCA package.

On your linux box where install failed, was it a clean python env? Or did it have a previous dPCA install? If it had a previous dPCA install, maybe go into your site-packages folder to make sure dPCA and UNKNOWN are wiped out before attempting to install.

cboulay commented 7 years ago

I'm on MacOS Sierra 10.12.2, using Python 3.6.0 installed with homebrew. My Python 2's numpy is broken so I can't test that.

sudo pip3 install -v does nothing because it needs a target to install. Output pasted anyway:

Chadwicks-MacBook-Pro:python chadwickboulay$ sudo pip3 install -v
Password:
The directory '/Users/chadwickboulay/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/chadwickboulay/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
You must give at least one requirement to install (see "pip help install")

If I add a . to the command ("sudo pip3 install -v ."), the output basically amounts to what I pasted above about not finding the include file.

Chadwicks-MacBook-Pro:python chadwickboulay$ sudo pip3 install -v .
The directory '/Users/chadwickboulay/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/chadwickboulay/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing /Volumes/STORE/Tools/Analysis/dPCA/python
  Running setup.py (path:/private/tmp/pip-ym7ws93r-build/setup.py) egg_info for package from file:///Volumes/STORE/Tools/Analysis/dPCA/python
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/dPCA.egg-info
    writing pip-egg-info/dPCA.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/dPCA.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/dPCA.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/dPCA.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/dPCA.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/dPCA.egg-info/SOURCES.txt'
    Compiling dPCA/nan_shuffle.pyx because it changed.
    [1/1] Cythonizing dPCA/nan_shuffle.pyx
    running egg_info
    creating pip-egg-info/UNKNOWN.egg-info
    writing pip-egg-info/UNKNOWN.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/UNKNOWN.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/UNKNOWN.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/UNKNOWN.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/UNKNOWN.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/UNKNOWN.egg-info/SOURCES.txt'
  Source in /private/tmp/pip-ym7ws93r-build has version 0.1, which satisfies requirement dPCA==0.1 from file:///Volumes/STORE/Tools/Analysis/dPCA/python
Installing collected packages: dPCA
  Running setup.py install for dPCA ...     Running command /usr/local/opt/python3/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-ym7ws93r-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-j2vnirdw-record/install-record.txt --single-version-externally-managed --compile
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/dPCA
    copying dPCA/__init__.py -> build/lib/dPCA
    copying dPCA/dPCA.py -> build/lib/dPCA
    running install_lib
    creating /usr/local/lib/python3.6/site-packages/dPCA
    copying build/lib/dPCA/__init__.py -> /usr/local/lib/python3.6/site-packages/dPCA
    copying build/lib/dPCA/dPCA.py -> /usr/local/lib/python3.6/site-packages/dPCA
    byte-compiling /usr/local/lib/python3.6/site-packages/dPCA/__init__.py to __init__.cpython-36.pyc
    byte-compiling /usr/local/lib/python3.6/site-packages/dPCA/dPCA.py to dPCA.cpython-36.pyc
    running install_egg_info
    running egg_info
    creating dPCA.egg-info
    writing dPCA.egg-info/PKG-INFO
    writing dependency_links to dPCA.egg-info/dependency_links.txt
    writing top-level names to dPCA.egg-info/top_level.txt
    writing manifest file 'dPCA.egg-info/SOURCES.txt'
    reading manifest file 'dPCA.egg-info/SOURCES.txt'
    writing manifest file 'dPCA.egg-info/SOURCES.txt'
    Copying dPCA.egg-info to /usr/local/lib/python3.6/site-packages/dPCA-0.1-py3.6.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-j2vnirdw-record/install-record.txt'
    running install
    running build
    running build_ext
    building 'dPCA.nan_shuffle' extension
    creating build/temp.macosx-10.12-x86_64-3.6
    creating build/temp.macosx-10.12-x86_64-3.6/dPCA
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c dPCA/nan_shuffle.c -o build/temp.macosx-10.12-x86_64-3.6/dPCA/nan_shuffle.o
    dPCA/nan_shuffle.c:444:10: fatal error: 'numpy/arrayobject.h' file not found
    #include "numpy/arrayobject.h"
             ^
    1 error generated.
    error: command 'clang' failed with exit status 1
error
Cleaning up...
  Removing source in /private/tmp/pip-ym7ws93r-build
Command "/usr/local/opt/python3/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-ym7ws93r-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-j2vnirdw-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-ym7ws93r-build/
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 878, in install
    spinner=spinner,
  File "/usr/local/lib/python3.6/site-packages/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "/usr/local/opt/python3/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-ym7ws93r-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-j2vnirdw-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-ym7ws93r-build/
wielandbrendel commented 7 years ago

Great (sorry, I mean "sudo pip3 install -v ." of course)! I am going to check your setup.py tomorrow (have to run) but one quick comment: there are now two setup(.) calls in setup.py, the second one without name, version, etc. which likely causes the UNKNOWN problem. I tested installation in a fresh Docker container.

cboulay commented 7 years ago

Right. The two setup() calls are how it's currently done in master. Even though I think there should only be one, I reverted back to this to help find out why it wasn't working for you.

Unfortunately my Linux machine is having some issues related to a video card upgrade that I don't have time to fix right now.

wielandbrendel commented 7 years ago

Alright, I tested the dPCA installation in a fresh Linux system and I am starting to get a clearer picture of what happened.

  1. I tested the wrong setup.py file yesterday because I forgot to checkout the correct branch. My bad, sorry!

  2. The setup.py in the current master seems to have been broken all along (e.g. using the two setup(.) calls). I probably didn't realise because my standard test - the toy example in jupyter notebook - worked fine but was just relying on the local dPCA code. Most people seem to have copied the dPCA script into their working directory, and so nobody complained so far.

  3. To get your modified setup.py script working properly on my Linux system I needed to join the two setup-calls into one (makes sense). Could you make this change in your branch?

  4. Could you add the package 'numexpr' as requirement. Pip can then install it automatically.

With these changes - and if you can confirm that the install works on your Mac - the PR is ready to roll :). Thanks for your efforts!

cboulay commented 7 years ago

I merged in latest master then made the requested changes to setup.py

wielandbrendel commented 7 years ago

Awesome, thanks @cboulay ! I'll add you to a to-be created contributor section in the Readme.