update build pipeline: simplify ci, move metadata into pyproject, support py312

tlambert03 commented 9 months ago

closes #37

I noticed that issue and spent some time updating the build pipeline here. Might have gotten a little carried away, but here's what this PR does now:

removes setup.cfg and places all metadata and setuptools config in pyproject.toml. The only thing setup.py is used for now is the c-extension stuff
uses cibuildwheel to build wheels on github actions: makes the action file significantly simpler. cibuildwheel config is in pyproject.toml
runs pytest on the built wheels
support python 3.12

an example build can be seen here: https://github.com/tlambert03/pymmcore/actions/runs/6484564479 ... and in addition to the pytest that gets run by cibuildwheel, I'll download those artifacts and test them on a couple real systems

This lets you build the project with python -m build ... or even with pip install -e . (assuming you have the build deps required)

@marktsuchida, if you're generally supportive of this, I can update the maintainer-notes.md to reflect all the changes made here.

things that could potentially be done as well:

remove smoketest, which is covered by the pytest
remove manylinux/build.sh

For the next deploy, this PR prepares deployment with pypi trusted publishers (rather than tokens) but @marktsuchida, you would need to set it up as described in that link

tlambert03 commented 9 months ago

also just noticed that you're currently building universal2 wheels, and this builds both x86_64 and arm64, let me know if you'd prefer me to switch the arm64 to a universal wheel

tlambert03 commented 9 months ago

To make it easier to discuss what wheels to keep/change, here's what got shipped for pymmcore v10.4.0.71.0:

cp311-win_amd64
cp311-win32
cp311-manylinux_2_17_x86_64.manylinux2014_x86_64
cp311-macosx_10_9_universal2
cp310-win_amd64
cp310-manylinux_2_17_x86_64.manylinux2014_x86_64
cp310-macosx_10_9_universal2
 cp39-win_amd64
 cp39-win32
 cp39-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp39-manylinux_2_17_i686.manylinux2014_i686
 cp39-macosx_10_9_x86_64
 cp38-win_amd64
 cp38-win32
 cp38-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp38-manylinux_2_17_i686.manylinux2014_i686
 cp38-macosx_10_9_x86_64

and in the artifacts from the last build in this PR, we have:

cp312-win_amd64
cp312-manylinux_2_17_x86_64.manylinux2014_x86_64
cp312-macosx_10_9_x86_64
cp312-macosx_11_0_arm64
cp311-win_amd64
cp311-manylinux_2_17_x86_64.manylinux2014_x86_64
cp311-macosx_10_9_x86_64
cp311-macosx_11_0_arm64
cp310-win_amd64
cp310-manylinux_2_17_x86_64.manylinux2014_x86_64
cp310-macosx_10_9_x86_64
cp310-macosx_11_0_arm64
 cp39-win_amd64
 cp39-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp39-macosx_10_9_x86_64
 cp39-macosx_11_0_arm64
 cp38-win_amd64
 cp38-manylinux_2_17_x86_64.manylinux2014_x86_64
 cp38-macosx_10_9_x86_64
 cp38-macosx_11_0_arm64

let me know what you want changed. (i suspect you might want win32 added back?)

marktsuchida commented 9 months ago

Fantastic! Thanks for knocking off a whole bunch of items from my to-do list (especially the use of ci-buildwheel)!

Would be great if you could update the maintainer notes (also feel free to move notes to source comments where things are now in one place, such as NumPy version selection; also feel free to remove any details that ci-buildwheel now takes care of). It is also fine to remove the smoke test and no-longer-needed Linux script.

There is no need to build Windows 32-bit wheels, since we no longer distribute 32-bit device adapters anyway.

I'm also happy with the single-arch wheels for macOS. The universal2 wheels were something of an accident anyway (I didn't explicitly do anything to build them). I think it is good to follow what NumPy (our only dependency) does when there is no reason to do otherwise, and NumPy currently ships 10_9_x86_64 and 11_0_arm64. Also I hope to ship arm64 Micro-Manager soon, and intend to do this as a separate download rather than trying to distribute fat binaries.

I'll review the details and the binary artifacts more carefully later but all the proposed directions look great.

marktsuchida commented 9 months ago

Thanks also for getting rid of the distutils imports. Another to-do item I had.

tlambert03 commented 9 months ago

ok, updated the notes. Let me know if it all makes sense

tlambert03 commented 9 months ago

i was able to test the artifacts on an intel mac and mac silicon (running the pymmcore-plus test suite)... haven't checked windows/linux yet