search

PyTables / PyTables

A Python package to manage extremely large amounts of data
http://www.pytables.org
BSD 3-Clause "New" or "Revised" License
1.29k stars 270 forks source link

setup.py doesn't install Numexpr #74

Closed hoffman closed 12 years ago

hoffman commented 13 years ago

Did python setup.py build earlier. I don't have Numexpr installed and running python setup.py install doesn't seem to install it, although I think it should.

hoffman@ebi-003 ~/sandbox/tables-2.2.1
$ python setup.py install
* Found numpy 1.5.1 package installed.
* Found HDF5 headers at ``/homes/hoffman/arch/Linux-x86_64/include``, library at ``/homes/hoffman/arch/Linux-x86_64/lib``.
.. WARNING:: Could not find the HDF5 runtime.
   The HDF5 shared library was *not* found in the default library
   paths. In case of runtime problems, please remember to install it.
* Could not find LZO 2 headers and library; disabling support for it.
* Could not find LZO 1 headers and library; disabling support for it.
* Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``.
running install
running build
running build_py
running build_ext
skipping 'tables/utilsExtension.c' Cython extension (up-to-date)
skipping 'tables/hdf5Extension.c' Cython extension (up-to-date)
skipping 'tables/tableExtension.c' Cython extension (up-to-date)
skipping 'tables/_comp_lzo.c' Cython extension (up-to-date)
skipping 'tables/_comp_bzip2.c' Cython extension (up-to-date)
cythoning tables/linkExtension.pyx to tables/linkExtension.c
building 'tables.linkExtension' extension
C compiler: gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O2 -march=nocona -pipe -fomit-frame-pointer -fPIC

compile options: '-DNDEBUG=1 -DHAVE_BZ2_LIB=1 -Iblosc -I/ebi/research/software/Linux_x86_64/opt/stow/python-2.7.1-shared/lib/python2.7/site-packages/numpy/core/include -I/homes/hoffman/arch/Linux-x86_64/include -I/nfs/acari/mh5/include/python2.5 -I/nfs/acari/mh5/include -I/ebi/research/software/Linux_x86_64/opt/stow/python-2.7.1-shared/include/python2.7 -c'
extra options: '-O2 -march=nocona -pipe -fomit-frame-pointer -Isrc -DH5_USE_16_API'
gcc: tables/linkExtension.c
tables/linkExtension.c: In function '__pyx_pf_6tables_13linkExtension_12ExternalLink_1_g_open':
tables/linkExtension.c:1824: warning: passing argument 4 of 'H5Lunpack_elink_val' from incompatible pointer type
tables/linkExtension.c:1824: warning: passing argument 5 of 'H5Lunpack_elink_val' from incompatible pointer type
tables/linkExtension.c: In function 'initlinkExtension':
tables/linkExtension.c:2698: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:2702: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:2706: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:2779: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:2779: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:2779: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:2779: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c: In function '__Pyx_PyObject_IsTrue':
tables/linkExtension.c:3810: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c:3811: warning: dereferencing type-punned pointer will break strict-aliasing rules
tables/linkExtension.c: At top level:
/ebi/research/software/Linux_x86_64/opt/stow/python-2.7.1-shared/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1188: warning: '_import_array' defined but not used
/usr/bin/gcc -pthread -shared -Wl,-rpath,/ebi/research/software/Linux_x86_64/opt/stow/python-2.7.1-shared/lib -Wl,-O1 -O2 -march=nocona -pipe -fomit-frame-pointer build/temp.linux-x86_64-2.7/tables/linkExtension.o -L/homes/hoffman/arch/Linux-x86_64/lib -L/ebi/research/software/Linux_x86_64/opt/stow/python-2.7.1-shared/lib -lhdf5 -lpython2.7 -o build/lib.linux-x86_64-2.7/tables/linkExtension.so
running build_scripts
running install_lib
copying build/lib.linux-x86_64-2.7/tables/_comp_lzo.so -> /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables
copying build/lib.linux-x86_64-2.7/tables/tableExtension.so -> /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables
copying build/lib.linux-x86_64-2.7/tables/utilsExtension.so -> /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables
copying build/lib.linux-x86_64-2.7/tables/linkExtension.so -> /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables
copying build/lib.linux-x86_64-2.7/tables/_comp_bzip2.so -> /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables
copying build/lib.linux-x86_64-2.7/tables/hdf5Extension.so -> /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables
running install_scripts
copying build/scripts-2.7/nctoh5 -> /homes/hoffman/arch/Linux-x86_64/bin
copying build/scripts-2.7/ptdump -> /homes/hoffman/arch/Linux-x86_64/bin
copying build/scripts-2.7/ptrepack -> /homes/hoffman/arch/Linux-x86_64/bin
changing mode of /homes/hoffman/arch/Linux-x86_64/bin/nctoh5 to 755
changing mode of /homes/hoffman/arch/Linux-x86_64/bin/ptdump to 755
changing mode of /homes/hoffman/arch/Linux-x86_64/bin/ptrepack to 755
running install_egg_info
Removing /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables-2.2.1-py2.7.egg-info
Writing /homes/hoffman/arch/Linux-x86_64/lib/python2.7/tables-2.2.1-py2.7.egg-info

hoffman@ebi-003 ~/sandbox/tables-2.2.1
$ cd ..

hoffman@ebi-003 ~/sandbox
$ python
Python 2.7.1 (r271:86832, Feb 28 2011, 15:37:40) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.__version__
'0.6c11'
>>> import numexpr
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numexpr
>>> import tables
*Fatal error*: You need Numexpr >= 1.4.1 for running PyTables!
scopatz commented 13 years ago

I think that installing numexpr separately isn't a huge deal. The goal is not write a package manager here. easy_install tables should install numexpr, if it is not installed already.

hoffman commented 13 years ago

setuptools should do this for us. I thought that PyTables used setuptools, but upon closer inspection of setup.py this is only if the FORCE_SETUPTOOLS environment variable is set. Do you know why it is done this way instead of using a try: ... except ImportError: ... block?

avalentino commented 13 years ago

Probably it is because using the environment variable allows to use standard distutils as default. See also:

http://groups.google.com/group/cython-users/browse_thread/thread/d6483338005ad91d

hoffman commented 13 years ago

This comment in setup.py seems to indicate that there is a workaround such that it should work even if setuptools is being used:

        # For some reason, setup in setuptools does not compile
        # Cython files (!) Do that manually...
avalentino commented 13 years ago

Yes, the get_cython_extfiles function tries to cythonize pyx files if the corresponding .c file doesn't exist or if the .c file is older than the pyx file (this should never happen if one uses the original tarball).

IMHO there are tho scenarios:

  1. pytables is installed from the source tarball, in this case cython is actually not needed since .c modules already are in the distribution
  2. the user get sources from the vcs or hacks the original tarball and than wants to install pytables. In this scenario IMHO it makes perfectly sense to assume that the user is skilled enough to know how to setup the development environment

In both cases IMHO the behavior is correct.

Of course it would be nice to have a setuptools based setup script properly working.

FrancescAlted commented 13 years ago

setuptools has traditionally be a PITA for me (IMO, and contrarily to what its blurb says, it does create more problems than it solves, at least from a developer point of view). Feel free to improve this support and good luck in the endeavour!

scopatz commented 13 years ago

+1 to Francesc. setuptools definitely creates more problems than it solves.

hoffman commented 13 years ago

It is a bit of a mess. easy_install makes it much easier for people (such as myself) to create other packages that rely on PyTables, so I hope that support will continue.

kcarnold commented 13 years ago

My research group also deals with this issue. For projects with many packages, we use setuptools to verify a hard-coded list of packages, falling back to distutils if setuptools is not present. (distutils just ignores 'install_requires'.) See, e.g., https://github.com/commonsense/simplenlp/blob/master/setup.py#L9

For projects with Cython extensions, our try-building-with-cython looks like https://github.com/commonsense/divisi2/blob/master/setup.py#L60. Theoretically we should be able to just use http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils but we haven't tried it.

avalentino commented 12 years ago

OK, there seems to be a general consensus on leaving setuptools out of our setup.py and use the standard Distutils. The user manual seems to be clear enough on how to install PyTables from sources, I will just a a note regarding installation from a git checkout.

Can we consider the issue closed?

scopatz commented 12 years ago

@avalentino, I think we can. Closing.