Support for Python 3.9

[amotl]

Dear @FrancescAlted, @robbmcleod and @pydata,

first things first: Thank you so much for conceiving and maintaining NumExpr.

Binary wheels for Python 3.9 for all of lxml, h5py, Pandas and Numpy are already available on PyPI these days. At @earthobservations, we just have been able to upgrade to their most recent releases on Wetterdienst and it works like a charm, see https://github.com/earthobservations/wetterdienst/pull/202#issuecomment-727190718.

For PyTables, we also just outlined the situation at https://github.com/PyTables/PyTables/issues/823.

It would be awesome if you could also build and upload cp39 artefacts to PyPI to make it easier for downstream packages and their users to upgrade to Python 3.9.

With kind regards, Andreas.

[robbmcleod]

I've tried looking into how to stage wheels through GitHub as Cython does it but I have not been able to figure out how to generate the API key.

https://raw.githubusercontent.com/MacPython/cython-wheels/master/.travis.yml

https://raw.githubusercontent.com/MacPython/cython-wheels/master/appveyor.yml

whereas we used to build from:

https://github.com/MacPython/numexpr-wheels

[amotl]

Dear Robert,

thanks for taking a look into this. Apologies for not knowing about any details behind the package publishing infrastructure, but might the update https://github.com/matthew-brett/multibuild/commit/26305eb32be5d33ce97f12a43966146ab4af1f34 by @matthew-brett regarding the switch from Rackspace only recently be related in any way?

Ah, I just recognized that you are already aware of this detail (https://github.com/matthew-brett/multibuild/issues/349). So, might the guidelines at https://github.com/matthew-brett/multibuild/issues/349#issuecomment-684060686 by @peterjc help eventually?

Ah, again - no. @peterjc was using Anaconda for staging wheels and you are looking at using GitHub for that very purpose, right? If that might be of any help, the respective discussion on cython-wheels probably was https://github.com/MacPython/cython-wheels/issues/3?

So, did you follow https://github.com/MacPython/cython-wheels/commit/523b8213f34b39c63d85fc42e2febe5600a59e5d and https://github.com/MacPython/cython-wheels/commit/ca2a09bd35bb45136502603a376026902c9f08c9 already? Both commits have been referenced within the discussion at https://github.com/MacPython/cython-wheels/issues/3. Maybe @scoder or @mattip are willing to help with the details?

Sorry again for the unqualified noise. From having a look behind the scenes of these aspects, we again appreciate the tremendous amount of work you are all doing here in order to bring Python packages for scientific computing to the general public.

Keep up the spirit and with kind regards, Andreas.

[robbmcleod]

After looking into it further I'm quite keen to switch to GitHub Actions instead of using Appveyor/Travis. I'll probably figure out how to do it first for my cpufeature package (so I don't spam all the watchers here with commits). If you want to follow along I've made a new issue to track it:

https://github.com/robbmcleod/cpufeature/issues/7

It looks like we'll be able to have separate actions for pushes versus releases, so tests and wheel builds all in this repo should be possible (making numexpr-wheels defunct).

[mattip]

A couple of hints: cibuildwheel is a nice way to build wheels, they support github actions among other options. You may want to think about an arm64 wheel: github actions is x86-only. NumPy uses travis-ci.com (since they also have ppc64 and s390x) and shippable.com (for arm64 only). And if you end up using cibuildwheel, a pypy wheel should JustWork. If not, please report a bug to pypy at https://foss.heptapod.net/pypy/pypy/-/issues

[robbmcleod]

@mattip Thanks for the advice. Looking at the cibuildwheel docs it suggests TravisCI can build arm64 so why is NumPy using shippable.com?

Do you know how one would go about getting 'preview' wheels from TravisCI for a sanity check before pushing a new tag if I were to use this script?

https://github.com/joerick/cibuildwheel/blob/master/examples/travis-ci-deploy.yml

[mattip]

~arm64: it uses qemu, not a native build. That is slow and adds another layer of indirection, which would be difficult to debug.~ Sorry, that was wrong. Travis does have native machines, in fact it has more powerful machines than shippable. NumPy uses shippable to spread out the work among different CI providers, which it turns out is a good thing because CI runs of SciPy and NumPy just ran out of credits on travis.

As for saving artifacts from the build: you can push non-tagged "developer" wheels to test.pypi.org

[Harmon758]

Note, without wheels, the underlying issue for #305, https://github.com/numpy/numpy/issues/11451, reoccurs with Python 3.9.

[robbmcleod]

So the wheel building process appears to be working:

https://github.com/pydata/numexpr/

It is building both manylinux1 and manylinux2010 wheels at present.

However, I did not specify a numpy version for each version. I forget if this can create binary issues or not if we link to a different version of the NumPy API that what a user has installed.

Previously the numpy version was specified in the numexpr-wheels:

https://github.com/MacPython/numexpr-wheels/blob/master/appveyor.yml

https://github.com/MacPython/numexpr-wheels/blob/master/.travis.yml

@matthew-brett or @mattip would either of you comment on this?

[mattip]

In general this should not be a problem unless you run afoul of some of the more obscure corners of NumPy. You could try it out by building a wheel with one version of numpy, then creating a virtual environment for testing, installing the wheel into that virtualenvironment while installing a different version of numpy, and running tests. The wheel-building framework you ~are~ were using over at MacPython/numexpr-wheels, multibuild, supports this via the NP_BUILD_DEP and NP_TEST_DEP environment variables.

[robbmcleod]

NumExpr 2.7.2 is released with Python 3.9 wheels, closing.