Raysect bdist wheel incompatible with older numpy

[jacklovell]

Steps to reproduce:

1. Create a new conda environment or virtualenv with Python 3.6 or 3.7.
2. pip install cython matplotlib numpy==1.15
3. pip install raysect
4. Attempt to use raysect.

Result:

jlovell@jlovell-thinkpad:~$ python -m unittest discover raysect
EEE
======================================================================
ERROR: raysect.core (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: raysect.core
Traceback (most recent call last):
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/unittest/loader.py", line 462, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/site-packages/raysect/core/__init__.py", line 30, in <module>
    from .ray import *
  File "raysect/core/math/_vec3.pxd", line 32, in init raysect.core.ray
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/site-packages/raysect/core/math/__init__.py", line 39, in <module>
    from .statsarray import StatsBin, StatsArray1D, StatsArray2D, StatsArray3D
  File "__init__.pxd", line 918, in init raysect.core.math.statsarray
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

======================================================================
ERROR: raysect.optical (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: raysect.optical
Traceback (most recent call last):
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/unittest/loader.py", line 462, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/site-packages/raysect/optical/__init__.py", line 53, in <module>
    from raysect.core.intersection import *
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/site-packages/raysect/core/__init__.py", line 30, in <module>
    from .ray import *
  File "stringsource", line 105, in init raysect.core.ray
AttributeError: type object 'raysect.core.ray.array' has no attribute '__reduce_cython__'

======================================================================
ERROR: raysect.primitive (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: raysect.primitive
Traceback (most recent call last):
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/unittest/loader.py", line 462, in _find_test_path
    package = self._get_module_from_name(name)
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/site-packages/raysect/primitive/__init__.py", line 30, in <module>
    from .box import Box
  File "raysect/primitive/box.pyx", line 1, in init raysect.primitive.box
  File "/home/jlovell/Software/anaconda3/envs/cherab-test/lib/python3.6/site-packages/raysect/core/__init__.py", line 30, in <module>
    from .ray import *
  File "stringsource", line 105, in init raysect.core.ray
AttributeError: type object 'raysect.core.ray.array' has no attribute '__reduce_cython__'

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (errors=3)

Looks like the Numpy ABI changed between 1.15 and 1.16, and the manylinux wheels are built with 1.16.4. However, the numpy dependency in setup.py is not given a minimum version, so the wheel is able to be installed on systems with numpy < 1.16 without complaint.

Building raysect from source with Numpy 1.15 works fine, so it doesn't seem necessary to build the wheel with 1.16. Updating Numpy to 1.18.1 and installing raysect using the wheel also works fine, so Numpy versions seem to be forward compatible. The wheel should therefore be built with the oldest numpy version possible, and that version set as the minimum for the Numpy dependency.

[jacklovell]

Raysect currently provides binary wheels for Python 3.6 and 3.7. The earliest Numpy bdist_wheel for 3.6 is 1.11.3, and the earliest bdist_wheel for 3.7 is 1.14.6. Is it reasonable to use one of these as the minimum Numpy version? If a user has an older Numpy than this for their Python version then we know they'll have had to build that from source anyway, so would presumably be willing to build raysect from source too.

[CnlPepper]

Hi Jack, I can't entirely remember why I chose 1.16, but I think there were issues using an earlier release. Unfortunately you can't (afaik) specify separate dependency versions for wheels. Pypi only considers the python version and architecture. Users of other versions should install from source.

[jacklovell]

I've made a manylinux wheel of 0.6 (from the development branch) with Numpy 1.12.1 on Python 3.6, which passes the unit tests and renders the couple of demos I've tried correctly. I've also tried 1.14.6 on Python 3.7, with the same results. Could we not just specify e.g. numpy>=1.14.6 in the install_requires in setup.py and build the wheel with this version of Numpy? Pip should then refuse to install raysect when an older Numpy is installed, unless Numpy is upgraded.

This issue actually arose when a user at UKAEA tried to do pip install raysect on the Freia cluster, which has Python 3.7 and Numpy 1.15.4. We were seeing some very strange errors (see above for the __reduce_cython__ red herring), and it was only by looking in raysect's development branch at the notes for buliding wheels that I could guess what Numpy version was used to build the wheel. At the very least there should be some specification of the supported Numpy version visible to end users trying to install the package, and I think install_requires is the most appropriate place for this.

[CnlPepper]

I'll have another look at it for the next release, if we can use 1.14.6 then I'll move to that and set the install_requires. I have a feeling it caused an issue for downstream packages e.g. cherab. Cherab and raysect have to be built against the same binary compatible versions of numpy otherwise it'll cause structure errors.

The whole wheel support of pip and pypi is a mess due to not being able to specify specific versions of libraries in each wheel. I'd like to set the wheel for numpy >= 1.16 but the source install for 1.14 if that fails.

[CnlPepper]

If you can come up with a recipe that works for raysect and cherab I'll move over to it.

[jacklovell]

I did the following. I used singularity rather than docker because I can't run docker as root on the CCFE clusters, but otherwise mostly followed the manylinux instructions. The gitrepos directory in the script below contains both raysect-source and cherab-core. The steps were:

1. Install raysect build dependencies (latest cython, numpy==1.14.6)
2. Build the raysect wheel
3. Install the raysect wheel
4. Build the cherab wheel

This was all done in the same container so that Cherab and Raysect were built against the same version of Numpy.

# Start container
singularity shell -c -w -B ~/gitrepos:/tmp docker://quay.io/pypa/manylinux2010_x86_64
# Install build dependencies
/opt/python/cp37-cp37m/bin/python -m pip install cython numpy==1.14.6
# Build Raysect wheel
cd /tmp/raysect-source
# Use build_ext to build with multiple cores, then package into a wheel
/opt/python/cp37-cp37m/bin/python setup.py build_ext -j$(nproc --all)
/opt/python/cp37-cp37m/bin/python setup.py bdist_wheel
auditwheel repair dist/raysect-0.6.1-cp37-cp37m-linux_x86_64.whl
# Install Raysect so Cherab can be built. Need to cd because the
# install gets skipped if there's a 'raysect' directory in CWD
cd /
/opt/python/cp37-cp37m/bin/python -m pip install /tmp/raysect-source/wheelhouse/raysect-0.6.1-cp37-cp37m-manylinux1_x86_64.whl
# Build Cherab wheel
cd /tmp/cherab-core
/opt/python/cp37-cp37m/bin/python setup.py build_ext -j$(nproc --all)
/opt/python/cp37-cp37m/bin/python setup.py bdist_wheel
auditwheel repair dist/cherab-1.2.1-cp37-cp37m-linux_x86_64.whl

Installing Raysect and Cherab from these manylinux1 wheels and running nosetests raysect and nosetests cherab worked on systems with Numpy 1.15.4 (SL7), 1.17.2 and 1.18.1 (both on Mint 19, an Ubuntu 18.04 derivative). I also tested on somebody else's user account on Freia and they worked on that too (they had not used raysect or cherab previously). I think the key is to ensure both Raysect and downstream packages like Cherab are built with the same as-old-as-possible Numpy version, or at least ABI-compatible Numpy versions. I think the ABI incompatibility was introduced in Numpy 1.16.0, so in theory building both projects with Numpys older than this should work. But if 1.14.6 is sufficient for both there's no good reason to use a newer version.

Also, I specified numpy>=1.14.6 in Raysect's setup.py, so if an older version of Numpy is already present then pip install raysect will update Numpy. For Cherab, I needed to modify setup.py to specify numpy>=1.14.6, and also raysect>=0.6 rather than raysect==0.6.0, as the Raysect development branch now advertises the version as 0.6.1 which is then built into the wheel.

[CnlPepper]

I've made a new release, v0.6.1. This has fixed the compatibility issue with numpy v1.14. The cherab development branch is now passing the tests in a v1.14 environment.