wheel not building on arm64 systems

[daltonv]

I am building a docker targeting arm64 systems and recently joulescope stop building correctly.

I am installing pyjoulescope v1.1.14 and I get the following error:

building 'joulescope.v0.stream_buffer' extension
#18 307.1       creating build/temp.linux-aarch64-cpython-310
#18 307.1       creating build/temp.linux-aarch64-cpython-310/joulescope
#18 307.1       creating build/temp.linux-aarch64-cpython-310/joulescope/v0
#18 307.1       creating build/temp.linux-aarch64-cpython-310/joulescope/v0/native
#18 307.1       aarch64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/pip-build-env-m3648div/overlay/local/lib/python3.10/dist-packages/numpy/_core/include -I/usr/include/python3.10 -c joulescope/v0/native/running_statistics.c -o build/temp.linux-aarch64-cpython-310/joulescope/v0/native/running_statistics.o
#18 307.1       aarch64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/pip-build-env-m3648div/overlay/local/lib/python3.10/dist-packages/numpy/_core/include -I/usr/include/python3.10 -c joulescope/v0/stream_buffer.c -o build/temp.linux-aarch64-cpython-310/joulescope/v0/stream_buffer.o
#18 307.1       In file included from /tmp/pip-build-env-m3648div/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/ndarraytypes.h:1909,
#18 307.1                        from /tmp/pip-build-env-m3648div/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
#18 307.1                        from /tmp/pip-build-env-m3648div/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/arrayobject.h:5,
#18 307.1                        from joulescope/v0/stream_buffer.c:1255:
#18 307.1       /tmp/pip-build-env-m3648div/overlay/local/lib/python3.10/dist-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#18 307.1          17 | #warning "Using deprecated NumPy API, disable it with " \
#18 307.1             |  ^~~~~~~
#18 307.1       In file included from /usr/include/python3.10/Python.h:74,
#18 307.1                        from joulescope/v0/stream_buffer.c:41:
#18 307.1       joulescope/v0/stream_buffer.c: In function ‘__pyx_f_5numpy_PyDataType_SHAPE’:
#18 307.1       joulescope/v0/stream_buffer.c:21949:39: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’
#18 307.1       21949 |     __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
#18 307.1             |                                       ^~
#18 307.1       /usr/include/python3.10/object.h:112:41: note: in definition of macro ‘_PyObject_CAST’
#18 307.1         112 | #define _PyObject_CAST(op) ((PyObject*)(op))
#18 307.1             |                                         ^~
#18 307.1       joulescope/v0/stream_buffer.c:2525:27: note: in expansion of macro ‘Py_INCREF’
#18 307.1        2525 |   #define __Pyx_INCREF(r) Py_INCREF(r)
#18 307.1             |                           ^~~~~~~~~
#18 307.1       joulescope/v0/stream_buffer.c:21949:5: note: in expansion of macro ‘__Pyx_INCREF’
#18 307.1       21949 |     __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
#18 307.1             |     ^~~~~~~~~~~~
#18 307.1       joulescope/v0/stream_buffer.c:21950:36: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘subarray’
#18 307.1       21950 |     __pyx_r = ((PyObject*)__pyx_v_d->subarray->shape);
#18 307.1             |                                    ^~
#18 307.1       error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
#18 307.1       [end of output]
#18 307.1   
#18 307.1   note: This error originates from a subprocess, and is likely not a problem with pip.
#18 307.1   Building wheel for patch-ng (setup.py): started
#18 307.1   ERROR: Failed building wheel for joulescope
#18 308.4   Building wheel for patch-ng (setup.py): finished with status 'done'
#18 308.4   Created wheel for patch-ng: filename=patch_ng-1.17.4-py3-none-any.whl size=16094 sha256=ddbd65ccbdf96bb07f054d29dc7a8fdf1b295e0e88b95924a0a69c9afc135039
#18 308.4   Stored in directory: /tmp/pip-ephem-wheel-cache-ed5cuhi6/wheels/57/3c/2b/10c0df622d24354a30a2304d4a9dfeefce998f24866aa27861
#18 308.4   Building wheel for psutil (pyproject.toml): started
#18 330.2   Building wheel for psutil (pyproject.toml): finished with status 'done'
#18 330.2   Created wheel for psutil: filename=psutil-5.9.8-cp310-abi3-linux_aarch64.whl size=287762 sha256=f7f732283ac995d16bf7c87ea3fd86fa8008f73fdca3460aa662bbd54ecb7579
#18 330.2   Stored in directory: /tmp/pip-ephem-wheel-cache-ed5cuhi6/wheels/3e/93/d6/85cd469d2103627a9e38acdccc834a9997e77d2abe6da25c8b
#18 330.2   Building wheel for pyjls (pyproject.toml): started
#18 346.8   Building wheel for pyjls (pyproject.toml): finished with status 'error'
#18 346.9   error: subprocess-exited-with-error

I'm guessing this is due to some dependency change with either numpy or gcc.

[daltonv]

I think I found the issue. Numpy released 2.0.0 a few weeks ago and it really does make a breaking change to the PyArray_Descr struct.

The pyproject.toml file does NOT pin the version of numpy in the build dependencies. This seems like it would be an issue for all arches but probably went unnoticed because prebuilt wheels are shipped for other arches.

[daltonv]

Looks like this does really only affect arm wheel builds. While they still get numpy 2.0.0 for a build dep cython is generating c code differently that does not access the non-existent member of PyArray_Descr.

[mliberty1]

The release of Numpy 2 is definitely causing issues. Our wheels built with numpy 1.x do not run with numpy 2.x. We distribute the cython generated .c in our source distributions, which was the standard recommendation to avoid requiring cython on package install. Unfortunately, the .c generated for numpy 1 is not compatible with numpy 2. While pyjls pins numpy 1, neither pyjoulescope_driver or joulescope do.

For now, you can try to manually install numpy 1.26.4:

pip install numpy==1.26

Does this work around the issue for now?

[daltonv]

That does not fix the issue. python already installs 1.26.4, but the build dependency is a separate install just inside the temporary build folder. I'm not sure I can force pip to use a specific version at that stage.

[daltonv]

Ah it works if I run pip with the --no-build-isolation. I just need to manually install numpy and wheel

[mliberty1]

I am also working to create releases of pyjls 0.9.6, pyjoulescope_driver 1.5.4, and joulescope 1.1.15 that all have the numpy version pinned to < 2. This will at least give us time to figure out a proper way to handle this numpy dependency challenge.

pyjls and pyjoulescope_driver are done. We just need to wait on the GitHub Action for the joulescope build: https://github.com/jetperch/pyjoulescope/actions/runs/9712756076

It publishes to pypi automatically, so it should just work once the GitHub Action completes.

[mliberty1]

The build completed, so you should automatically get 1.5.4 with numpy < 2. Does the joulescope package now build correctly on linux aarch64 for you?

[daltonv]

My build just finished. Everything is working again. Thanks for the super quick fix!