Failing to build manylinux2014 AArch64 wheels

[odidev]

On AArch64 machine, pip installl imagecodecs tries to build the wheels from source code and fails with below error-

    /opt/_internal/cpython-3.8.12/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
       17 | #warning "Using deprecated NumPy API, disable it with " \
          |  ^~~~~~~
    imagecodecs/_blosc2.c:627:10: fatal error: blosc2.h: No such file or directory
      627 | #include "blosc2.h"
          |          ^~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1

@cgohlke, can you please let me know your interest in deploying AArch64 wheels on PyPI? Can you please point out your wheel builder? I would like to help in this.

[cgohlke]

Looks like you have LD_LIBRARY_PATH or LIBRARY_PATH defined in your environment. Please confirm. Otherwise blosc2 would not be required.

Please see the Requirements and Notes for building from source. The Linux and macOS wheels are built based on @Czaki https://github.com/Czaki/imagecodecs_build repro so you may want to contribute there.

[Czaki]

The tool which I use for build wheels (cibuildwheel) support now aarch64. I do not have access to aarh64 machine so I'm fully open on cooperation to add its support with someone with such hardware acces.

[cgohlke]

Thanks @Czaki . Somewhat related to this issue: do you know the status of Apple silicon in cibuildwheel and Azure Pipelines?

[Czaki]

Also supported. Also lack of access to hardware. I could try to enable but macos 11.0 introduce some changes in security mechanism, which breaks build process. With cooperation of one person I start fixing it but to finish it in a relatively short time there is a need to have access to m1 hardware. (Building double architecture libraries).

Or maybe simpler it will be to build pure arm wheels, not universal2 ones.

[cgohlke]

Looks like you have LD_LIBRARY_PATH or LIBRARY_PATH defined in your environment. Please confirm. Otherwise blosc2 would not be required.

https://github.com/cgohlke/imagecodecs/blob/8eec683f117b44c178785f542831f530511608ff/setup.py#L509-L510

It's probably better to define a IMAGECODECS_BUILD environment variable in the CI and use that instead of LD_LIBRARY_PATH or LIBRARY_PATH.

[Czaki]

It's probably better to define a IMAGECODECS_BUILD environment variable in the CI and use that instead of LD_LIBRARY_PATH or LIBRARY_PATH.

Agree. I will switch.

I prefer to wait for https://github.com/pypa/cibuildwheel/pull/854 to be merged and then move the configuration to pyproject.toml which will simplify live and allow easier select CI provider.

[henryiii]

You can use cibuildwheel.toml and --config=cibuildwheel.toml then. Though I would recommend trying to learn what it actually does and you'll find it's fantastic and solves important problems. Also, don't delete pyproject.toml, just use --no-build-isolation or PIP_NO_BUILD_ISOLATION=<x> (never remember how you set a "no" variable in Pip, other than it's weird and backward. Maybe 0 or off or something like that).

[henryiii]

I have also had problems with pyproject.toml, but the problem is always that they are missing from some dependencies. ;)

[Czaki]

I was thinking to move the configuration to any toml file. Building aarch64 is supported by emulation it consumes a long time so it may be nice to move some of the work on GitHub actions.

[odidev]

@Czaki, How about using CircleCI to build aarch64 wheels? CircleCI provides free AArch64 build agent.

[Czaki]

I will try

[odidev]

Hi @Czaki, I was trying to build imagecodecs on AArch64 but it was failing while build jpeg-xl. I disabled same and now build is successful. After that I executed the tests and again only one test was failing which was related to jpeg-xl. I added a patch(which can be removed and code changes can be shipped to actual imagecodec repository) to xfail that particular test for AArch64 machine. Now build and tests are successful. Also, I added the support for CircleCI.

Patch: https://github.com/odidev/imagecodecs_build/commit/62c03b0fb22daa2e7e0d50ec0366995f30e300bc Build logs: https://app.circleci.com/pipelines/github/odidev/imagecodecs_build/19/workflows/f2cbb2e1-d891-4a84-a922-ef1df078b54e/jobs/51 Artifacts: https://app.circleci.com/pipelines/github/odidev/imagecodecs_build/19/workflows/f2cbb2e1-d891-4a84-a922-ef1df078b54e/jobs/51/artifacts

Would you like to see a PR with the above changes?

[Czaki]

Hi, I found two issues anout jpeg-xl build failture https://github.com/libjxl/libjxl/issues/213 and https://github.com/libjxl/libjxl/issues/698. So I think that there is a chance that it will be working in future. @cgohlke Could you check this patch and decide if you prefer such changes or add special flag for this?

@odidev I will be happy with your PR. I only prefer to include your patch to main repository instead of applying it each time. But this decision need made @cgohlke

[cgohlke]

I only prefer to include your patch to main repository instead of applying it each time.

Sure. The aarch64.patch will be in the next release.

[Czaki]

Sure. The aarch64.patch will be in the next release.

it will be available in few days? It will simplify adding aarch64 support to the building repository.

[cgohlke]

it will be available in few days?

No, sorry.

[cgohlke]

The v2024.1.1 release includes manylinux_aarch64 wheels.