search

bccp / nbodykit

Analysis kit for large-scale structure datasets, the massively parallel way
http://nbodykit.rtfd.io
GNU General Public License v3.0
110 stars 60 forks source link

Current release (0.3.15) doesn't build in python 3.6 ? #688

Open dstndstn opened 1 month ago

dstndstn commented 1 month ago

Describe the bug

pip install nbodykit[extras] produces a ton of error messages, and ends up installing versions:

Successfully installed Corrfunc-2.5.0 bigfile-0.1.51 cached-property-1.5.2 cachey-0.2.1 coverage-6.2 dask-2021.3.0 future-1.0.0 h5py-3.1.0 heapdict-1.0.1 iniconfig-1.1.1 kdcount-0.3.25 mpmath-1.3.0 mpsort-0.1.18 nbodykit-0.2.5 numexpr-2.8.1 pluggy-1.0.0 pmesh-0.0.21 py-1.11.0 pytest-7.0.1 runtests-0.0.28 sympy-1.9 tomli-1.2.3 wurlitzer-3.1.1

Note that it has installed nbodykit-0.2.5.

Steps To Reproduce

# cat /etc/lsb-release  | grep RELEASE
DISTRIB_RELEASE=20.04

# pip --version
pip 21.2.2 from /cm/shared/apps/python/python36/lib/python3.6/site-packages/pip (python 3.6)

# pip install --upgrade nbodykit[extras] > log2 2>&1 &

Actual behaviour

Old version of nbodykit gets installed.

Expected behaviour

Newest version of nbodykit should get installed.

Version

heh,

# python -c 'import nbodykit; print(nbodykit, nbodykit.__version__);'
No protocol specified
--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[51580,1],0] (PID 2969345)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cm/shared/apps/python/python36/lib/python3.6/site-packages/nbodykit/__init__.py", line 7, in <module>
    dask.set_options(get=dask.get)
AttributeError: module 'dask' has no attribute 'set_options'
root@mn003:~# python -c "import nbodykit; print(nbodykit, nbodykit.__version__);"
No protocol specified
--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[51512,1],0] (PID 2969413)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cm/shared/apps/python/python36/lib/python3.6/site-packages/nbodykit/__init__.py", line 7, in <module>
    dask.set_options(get=dask.get)
AttributeError: module 'dask' has no attribute 'set_options'

If you suspect the problem is related to a dependency (e.g. Corrfunc, pmesh, or mcfit), then it is useful to report the version of that dependency as well (replacing nbodykit with the package name).

Additional context Provide any additional context about the problem here.

dstndstn commented 1 month ago

kdcount-0.3.29 fails with https://github.com/rainwoodman/kdcount/issues/18

Pip then tries kdcount-0.3.28 (same error), kdcount-0.3.27 (same error), kdcount-0.3.26 (same error), and then kdcount-0.3.25 (works, from wheel).

classylss-0.2.9 fails with:

Collecting classylss>=0.2
  Using cached classylss-0.2.9.tar.gz (244 kB)
    ERROR: Command errored out with exit status 1:
     command: /cm/shared/apps/python/python36/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-tz2ubhp4/classylss_479ed0f87def40ebb1ad204bac58ab5d/setup.py'"'"'
; __file__='"'"'/tmp/pip-install-tz2ubhp4/classylss_479ed0f87def40ebb1ad204bac58ab5d/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'f
rom setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-s45hvcem
         cwd: /tmp/pip-install-tz2ubhp4/classylss_479ed0f87def40ebb1ad204bac58ab5d/
    Complete output (2155 lines):
    /cm/shared/apps/python/python36/lib/python3.6/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from ear
lier releases! File: /tmp/pip-install-tz2ubhp4/classylss_479ed0f87def40ebb1ad204bac58ab5d/classylss/binding.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    from classylss import get_data_files

    _DATA_FILES = get_data_files()

    from cclassy cimport *
    ^
    ------------------------------------------------------------

    classylss/binding.pyx:12:0: 'cclassy.pxd' not found

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    def val2str(val):
        if isinstance(val, (list, tuple, np.ndarray)):
            return ','.join([str(i) for i in val])
        return str(val)

    cdef int _build_file_content(pars, file_content * fc) except -1:
                                       ^
    ------------------------------------------------------------

    classylss/binding.pyx:63:35: 'file_content' is not a type identifier

... and many many more errors.

Pip then tries classyls-0.2.8 (fails), classylss-0.2.7 (fails), all the way to classylss-0.2.1 (fails).

Pip then says

Collecting nbodykit[extras]
  Using cached nbodykit-0.3.14.tar.gz (302 kB)
Collecting sympy<=1.5.1
  Using cached sympy-1.5.1-py2.py3-none-any.whl (5.6 MB)
Collecting nbodykit[extras]
  Using cached nbodykit-0.3.13.tar.gz (301 kB)
  Using cached nbodykit-0.3.12.tar.gz (301 kB)
  Using cached nbodykit-0.3.11.tar.gz (301 kB)
  Using cached nbodykit-0.3.10.tar.gz (299 kB)
  Using cached nbodykit-0.3.9.tar.gz (298 kB)
  Using cached nbodykit-0.3.8.tar.gz (297 kB)
  Using cached nbodykit-0.3.7.tar.gz (291 kB)
  Using cached nbodykit-0.3.6.tar.gz (281 kB)
  Using cached nbodykit-0.3.5.tar.gz (280 kB)
  Using cached nbodykit-0.3.4.tar.gz (279 kB)
  Using cached nbodykit-0.3.3.tar.gz (274 kB)
  Using cached nbodykit-0.3.2.tar.gz (274 kB)
  Using cached nbodykit-0.3.1.tar.gz (267 kB)
  Using cached nbodykit-0.3.0.tar.gz (266 kB)
  Using cached nbodykit-0.2.9.tar.gz (244 kB)
  Using cached nbodykit-0.2.8.tar.gz (227 kB)
  Using cached nbodykit-0.2.7.tar.gz (220 kB)
  Using cached nbodykit-0.2.6.tar.gz (208 kB)
  Using cached nbodykit-0.2.5.tar.gz (200 kB)
Collecting h5py
...

so maybe it's backing off to a version of nbodykit that doesn't required classylss, or something like that?

sbird commented 1 month ago

First bug report as a new maintainer! My guess is that you are correct that it is classylss that is the problem. As a quick fix you could try installing https://github.com/sbird/classylss by hand using python -m build . Medium-term we are going to port nbodykit to regular classy.

dstndstn commented 1 month ago

Hi Simeon,

Thanks for your reply! Building classylss succeeds, apparently despite a couple of bash complaints:

$ python -m build .
...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython, numpy, setuptools>=42, wheel)
* Getting build dependencies for wheel...
Compiling classylss/binding.pyx because it depends on /tmp/build-env-2ju1h15j/lib/python3.6/site-packages/Cython/Includes/libc/string.pxd.
[1/1] Cythonizing classylss/binding.pyx
warning: classylss/binding.pyx:615:112: Use boundscheck(False) for faster access
warning: classylss/binding.pyx:618:38: Use boundscheck(False) for faster access
performance hint: classylss/binding.pyx:1447:32: Exception check after calling 'pk' will always require the GIL to be acquired. Declare 'pk' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
running egg_info
writing classylss.egg-info/PKG-INFO
writing dependency_links to classylss.egg-info/dependency_links.txt
writing requirements to classylss.egg-info/requires.txt
writing top-level names to classylss.egg-info/top_level.txt
reading manifest file 'classylss.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'classylss.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/classylss
copying classylss/__init__.py -> build/lib.linux-x86_64-3.6/classylss
copying classylss/astropy_compat.py -> build/lib.linux-x86_64-3.6/classylss
copying classylss/version.py -> build/lib.linux-x86_64-3.6/classylss
running build_ext
curl -fsSL https://github.com/lesgourg/class_public/archive/v2.7.2.tar.gz \
    > class-v2.7.2.tar.gz
mkdir -p tmp-class-v2.7.2
gzip -dc class-v2.7.2.tar.gz | tar xf - -C tmp-class-v2.7.2
touch tmp-class-v2.7.2/class_public-2.7.2/stamp.unzip
for i in tmp-class-v2.7.2/class_public-2.7.2/stamp.unzip 0001-class-2.6.0-a_max.patch 0002-class-2.6.1-pathsize.patch 0003-Add-FLT_EPSILON-to-array_interpolate_spline.patch; do echo $i; \
if [ "${i%%.patch}" != $i ]]; then \
    patch -d tmp-class-v2.7.2/class_public-2.7.2 -p1 < $i || exit 1; \
fi; \
done; \
touch tmp-class-v2.7.2/class_public-2.7.2/stamp.patch
tmp-class-v2.7.2/class_public-2.7.2/stamp.unzip
/bin/sh: line 1: [: missing `]'
0001-class-2.6.0-a_max.patch
/bin/sh: line 1: [: missing `]'
0002-class-2.6.1-pathsize.patch
/bin/sh: line 1: [: missing `]'
0003-Add-FLT_EPSILON-to-array_interpolate_spline.patch
/bin/sh: line 1: [: missing `]'
cp Makefile.class tmp-class-v2.7.2/class_public-2.7.2/Makefile
cp class.cfg tmp-class-v2.7.2/class_public-2.7.2/myclass.cfg
cd tmp-class-v2.7.2/class_public-2.7.2; make CLASSCFG=myclass.cfg libclass.a
...
Successfully built classylss-0.2.10.dev0.tar.gz and classylss-0.2.10.dev0-cp36-cp36m-linux_x86_64.whl

I then pip-installed the resulting dist/*.whl file, then did

# pip install --upgrade nbodykit[extras]
...
Requirement already satisfied: classylss>=0.2 in /cm/shared/apps/python/python36/lib/python3.6/site-packages (from nbodykit[extras]) (0.2.10.dev0)
...
Successfully installed mcfit-0.0.21 nbodykit-0.3.15

So that did the trick! Thanks again!

sbird commented 1 month ago

Glad to hear it! I will leave this open until pip install nbodykit just works without issues.