Closed adler-j closed 8 years ago
Looking good up to minor things. I'll try to build and install myself first though.
I didn't look too closely on the doc since I'm anyway doing a big overhaul. Design-wise it looks good.
Okay, after some playing around and observing that conda
tries to be smart and not to copy stuff around more than it has to, I finally got a freshly built odlpp
installed. When trying to import it, I get
File "/home/hkohr/miniconda3/envs/odlpp_test_27/lib/python2.7/site-packages/odlpp/__init__.py", line 23, in <module>
from . import odlpp_cuda
ImportError: cannot import name odlpp_cuda
Looking at the output from the build process, it's no surprise:
...
[100%] Linking CXX shared library ../../bin/libodlpp_cuda.so
[100%] Built target odlpp_cuda
Scanning dependencies of target pyinstall
running install
running bdist_egg
running egg_info
creating odlpp.egg-info
writing requirements to odlpp.egg-info/requires.txt
writing odlpp.egg-info/PKG-INFO
writing top-level names to odlpp.egg-info/top_level.txt
writing dependency_links to odlpp.egg-info/dependency_links.txt
writing entry points to odlpp.egg-info/entry_points.txt
writing manifest file 'odlpp.egg-info/SOURCES.txt'
reading manifest file 'odlpp.egg-info/SOURCES.txt'
writing manifest file 'odlpp.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/odlpp
copying ./odl_pluggin.py -> build/lib/odlpp
copying ./cu_ntuples.py -> build/lib/odlpp
copying ./__init__.py -> build/lib/odlpp
copying ./ufuncs.py -> build/lib/odlpp
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/odlpp
copying build/lib/odlpp/odl_pluggin.py -> build/bdist.linux-x86_64/egg/odlpp
copying build/lib/odlpp/cu_ntuples.py -> build/bdist.linux-x86_64/egg/odlpp
copying build/lib/odlpp/__init__.py -> build/bdist.linux-x86_64/egg/odlpp
copying build/lib/odlpp/ufuncs.py -> build/bdist.linux-x86_64/egg/odlpp
byte-compiling build/bdist.linux-x86_64/egg/odlpp/odl_pluggin.py to odl_pluggin.pyc
byte-compiling build/bdist.linux-x86_64/egg/odlpp/cu_ntuples.py to cu_ntuples.pyc
byte-compiling build/bdist.linux-x86_64/egg/odlpp/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/odlpp/ufuncs.py to ufuncs.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying odlpp.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying odlpp.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying odlpp.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying odlpp.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying odlpp.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying odlpp.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
odlpp.__init__: module references __file__
creating dist
creating 'dist/odlpp-0.2.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing odlpp-0.2.0-py2.7.egg
creating /home/hkohr/miniconda3/envs/_build/lib/python2.7/site-packages/odlpp-0.2.0-py2.7.egg
Extracting odlpp-0.2.0-py2.7.egg to /home/hkohr/miniconda3/envs/_build/lib/python2.7/site-packages
Adding odlpp 0.2.0 to easy-install.pth file
Installed /home/hkohr/miniconda3/envs/_build/lib/python2.7/site-packages/odlpp-0.2.0-py2.7.egg
The compiled stuff is never touched and does not end up in the final package. Thus, only a skeleton package is installed.
I'm not quite sure why it is like that. Possible explanations are (1) the CMake commands never copy files or (2) conda
wraps up the package before the libraries are copied or ignores them in some other way.
I can build the code by hand to exclude reason (1). If it's related to (2), then probably we really need a setup.py
based build so conda
will see all the output, or some other way to manually copy the compiled files.
Same output for building "by hand", library files are not copied.
Absolutely no idea why it would not do that, it does the copy on windows but I had some issues with it (see the __path__
thing). Could you try checking the package_data
and see if you can get it to work on linux?
If I replace the package_data
in odlpp/setup.py
by package_data={'' : ['*.pyd', '*.so']}
, the files are included, and things finally work. Testing the conda build now.
¯_(ツ)_/¯
Great!
I think the proper way of doing this is to create a MANIFEST.in
file (see here) that contains all files apart from Python scripts/modules that are to be included in the distribution.
Added a MANIFEST.in and it is actually used during the setup.py install
command. However, it does not find the README.md and LICENSE files since they are not copied to the build directory. Here are the contents of build/bin
where python setup.py install
is run:
build dist libodlcpputils_cuda.a libodlcpputils_test.a MANIFEST.in odlpp_cuda.so setup.py
cu_ntuples.py __init__.py libodlcpputils_python.a libodlpp_cuda.so odl_pluggin.py odlpp.egg-info ufuncs.py
I know too little about CMake to figure out why it behaves like this. Perhaps the Stupid, I created the file in .in
suffix makes it believe it's a config file or similar and copies it by default.build/bin
and copied it over. Some way to make sure that also the other all metadata files are copied would be good.
Some way to make sure that also the other all metadata files are copied would be good.
Ping. Shouldn't be a big thing and would make this a working package.
TODOs:
MANIFEST.in
(and that file itself) are copied to the location where setup.py install
is run by make pyinstall
. Suggestion: pyinstall -> install
odl_cuda
or similar?setup.py
file (see the one in ODL for a template)bld.bat
for conda Windows builds. This should be possible starting from the build.sh
for Unix. One can use a similar approach with environment variables such that dependencies can be found. Should be done in the conda_build
branch though.EDIT: added main TODO to TODO list.
I'll try to get this done once I get back to KTH after midsummer, agree that it should be prio and we need to get it in rather soon. It would also enable the conda branch.
The checklist is very nicely compiled and should cover most things.
Regarding conda however, having a single branch for linux, getting that merged (which seems close?) and then making a new branch for windows would be totally fine.
Regarding conda however, having a single branch for linux, getting that merged (which seems close?) and then making a new branch for windows would be totally fine.
Sure.
Any chance you can push your changes to the branch so I can keep working from there?
If you don't need conda stuff, simply cherry-pick 71db46cd. It contains the MANIFEST.in file. No further relevant changes.
Work in progress.
Changes makes this a plugin instead of a subpackage.