CI and building for PyPy

[mattip]

Over at conda-forge they have started rolling out support for pypy3.7, and the CI run hangs. Have you considered running CI here? github actions would be a good candidate since they offer a nice recipe for python versions including pypy (probably best to test for pypy3.7, 3.6 is not long for the world). If you want to go all out and deploy from the repo, the cibuildwheel project builds, tests, and deploys for all the chosen python versions (https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip)in one run per platfom.

Edit: fix link to conda-forge issue

[cgohlke]

Coincidentally, I have just fixed test errors on PyPy for Windows this evening. The issue is/was that imagecodecs uses Cython memoryviews const uint8_t[::1] to write to the output of PyBytes_FromStringAndSize, which does not work on pypy. It could be the reason for the hangs. For now I have changed imagecoddecs to return bytearray instead of bytes on PyPy. The azure pipeline to build linux and macos wheels is at https://github.com/Czaki/imagecodecs_build. It has pypy enabled.

[mattip]

Cool, thanks. Should I close this then?

[Czaki]

I build pypy wheels for manylinux, but building wheels for macos fails because fail of installation on macos. numpy does not provide pypy wheel for pypy3.7.

    RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, see site.cfg.example for information. Otherwise report this to the vendor that provided NumPy.
    RankWarning: Polyfit may be poorly conditioned

I try to install openblas from brew but it happens again (I restart the job). @mattip maybe you have some suggestions on how to workaround this?

[mattip]

This comment suggests

brew install openblas
OPENBLAS="$(brew --prefix openblas)" pip install numpy

[Czaki]

It works. I will wait for fixed verion of imagecodecs. But @mattip in my case test fails, not stuck.

[mattip]

But @mattip in my case test fails, not stuck.

Can you point me to the log of a faiiled PyPy build? The one I found is still using accelerate for the PyPy macos run.

[mattip]

@cgohlke

Coincidentally, I have just fixed test errors on PyPy for Windows this evening

Is that change reflected in the repo here? Is it specifically for windows?

[Czaki]

You have stucked test on both macos and linux. In my case I think about linux one https://dev.azure.com/bokota/imagecodecs/_build/results?buildId=601&view=logs&jobId=b50d5a70-5aa0-546a-0931-d41e5c96779d

On MacOS I need wait about 50 minutes to check https://dev.azure.com/bokota/imagecodecs/_build/results?buildId=604&view=results

What you mean by using accelerate?

[Czaki]

There is also no stuck for macos job.

[hmaarrfk]

I guess you have a hidden branch or something. If you wanted to share the patch we could try applying to conda-forge to see if we are still experiencing the issues.

[Czaki]

The only patch which I have is connected with new clang version which cause fail on undeclared functions https://github.com/Czaki/imagecodecs_build/blob/master/imagecodecs.patch.

But maybe it is connected with used library vesions: https://github.com/Czaki/imagecodecs_build/blob/master/build_utils/download_libraries.sh

PyPy build in on master branch https://dev.azure.com/bokota/imagecodecs/_build/results?buildId=605&view=results

[Czaki]

You could also add pytest-timeout to test call. Then you will exactly know which test stuck.

[cgohlke]

Fixed in v2021.1.11

[hmaarrfk]

thanks!