search

project-gemmi / gemmi

macromolecular crystallography library and utilities
https://project-gemmi.github.io/
Mozilla Public License 2.0
228 stars 45 forks source link

pypi wheel builds? #42

Closed Anthchirp closed 3 years ago

Anthchirp commented 4 years ago

Could we have wheel releases on pypi? I am currently setting up CI builds for a new project which will be using gemmi, and it looks like building gemmi takes about 4 minutes, which adds up to some significant time with a build matrix of 4 python environments on 3 platforms.

dkratzert commented 4 years ago

That would be very nice. I could use it, too.

wojdyr commented 4 years ago

I suppose it'd be a significant work to set it up. If I'm wrong and it's relatively easy to do then yes, why not.

BTW, we have conda-forge builds maintained by Matt Swain.

JBGreisman commented 4 years ago

I agree this would be very useful for speeding up CI for my project as well -- building gemmi from source is currently the rate-limiting step. To my knowledge, this is a very straight-forward thing to add. When building your package for release, you can run:

python setup.py sdist bdist_wheel
twine upload dist/*

This should create 3 files in the dist directory: a tar.gz, a .whl, and a .egg, and upload them to your PyPI channel for gemmi. The gemmi PyPI page currently only has a tar.gz, so I think all that is needed it so add the bdist_wheel command to the setup.py call.

Anthchirp commented 4 years ago

Yes, that should be sufficient unless you need different binary builds for different python versions. (I don't know anything about the gemmi build process)

sizmailov commented 4 years ago

@JBGreisman It's not that simple for C++ extensions. @Anthchirp is correct, number of wheels needed for each supported python version/target OS/architectures. For example look at numpy pypi download page to realize the amount of work.

Note also that default bdist_wheel you get on linux is not manylinux.

JBGreisman commented 4 years ago

good point -- I had forgotten about supporting multiple Python versions / OS / architectures.

wojdyr commented 4 years ago

Looking at files from conda forge, they have 9 packages: python 3.6, 3.7 and 3.8 for Linux, Mac, Windows (all 64-bit).

For Linux, I suppose it needs to be built on the oldest supported Linux system to be portable (more specifically, the versions of glibc and C++ standard library would need to be old).

sizmailov commented 4 years ago

I think the situation is worse for pypi than for conda since bare pip lives in uncontrolled evnironment. Conda builds it's own python and other libraries so environments are under control and its number is quite low. I think it's almost impossible to make a pypi wheels for every combination exist. A meaningful compromise would be support of wheels on major systems and fall back to build-from-source on others.

Anthchirp commented 4 years ago

That is correct, which is what manylinux1, manylinux2010, manylinux2014 aim to address. I don't know what your current build/deploy process is, but the linked project has docker images that can be used to build wheels on travis as well as a link to a demo project.

wojdyr commented 4 years ago

Yes, that's the work I'd like to avoid: setting it up separately for each system and maintaining this setup. I have CI builds on Travis and AppVeyor and I suppose the wheels could be built there, but it'd still be an additional thing to setup and maintain.

I found an interesting project: conda-press - it creates wheels from conda packages. This would be the way to go, but apparently this project was actively developed only for half a year in 2019. From looking at the issues it doesn't support python3.8 packages yet.

wojdyr commented 4 years ago

I've uploaded Windows wheels for Python 3.6-3.8: https://pypi.org/project/gemmi/0.4.1/#files

They were built on AppVeyor: https://ci.appveyor.com/project/wojdyr/gemmi/builds/34355010 I was going to build also wheels for Python2.7, but it didn't work. I suppose I'd need to setup environment for VC++ or use a different worker image. Anyway, it's probably not important due to Python2 EOL.

dkratzert commented 4 years ago

Great, thank you! That is really helpful.

wojdyr commented 4 years ago

Now I added also Linux wheels: https://pypi.org/project/gemmi/0.4.2/#files They were built in docker image quay.io/pypa/manylinux2010_x86_64

Anthchirp commented 4 years ago

Using the new wheels reduces our Linux testing time from 5 minutes 55 seconds to 0 minutes 55 seconds. Massive impact. Thank you!

kmdalton commented 4 years ago

Linux builds are a big help! Thanks, @wojdyr

sizmailov commented 3 years ago

I suppose issue is not considered [completely] solved, so let me share my recent finding.

The cibuildwheel package can build wheels for wide range of platforms / architectures / python versions on many CI platforms. The list of examples contains pybind11-based project.

In fact, https://github.com/pybind/python_example has tested configuration files for Appveyor, Travis and Github-Actions.

wojdyr commented 3 years ago

Looks great! I'll give it a try soon. Thanks!

wojdyr commented 3 years ago

It's indeed much easier with cibuildwheel. I just made a new release and uploaded 31 wheels: https://pypi.org/project/gemmi/0.4.4/#files All were built in Github Actions: https://github.com/project-gemmi/gemmi/runs/1703771115

wojdyr commented 2 years ago

The latest cibuildwheels defaults to manylinux2014 image for Linux builds, previously it was manylinux2010. If anyone uses wheels on Linux distributions older than RHEL7 please let me know.

wojdyr commented 2 years ago

I'm wondering if it's time to stop testing and building wheels for Python 2.7. If anyone still needs versions older than 3.6 please let me know.