wheels should refuse to build if universal=1 but c exts are present

[zzzeek]

as stated on http://pythonwheels.com/:

Note: If your project is python 2 and 3 compatible you can create a universal wheel distribution. Create a file called setup.cfg with the following content and upload your package.

[wheel] universal = 1

Note: _this is not the solution for packages that have C extensions or other cases where a platform/architecture independent wheel is impossible. _

great, then instead of maintainers putting up useless wheels on pypi and creating confusion, have the wheel builder simply fail when the flag + C extensions are present, rather than causing confusion by building a non-universal wheel file. Per Nick Coghlan (https://twitter.com/ncoghlan_dev/status/437415973943922688):

"Hence why the long term plan is to eventually offer an automated build farm. Going to take a (long) while to get there, though."

what this means is, "wheels aren't ready". I don't understand why we're being bothered with them until they actually solve a problem.

sorry for the rant! just realized nobody's been running SQLAlchemy with C exts for a couple of months, since pip favors the wheel file over the source.

[dstufft]

The note you copied says that it's for pure python packages that are source compatible across 2 and 3.

Wheels are ready for most of their use cases and they do solve problems today. Just because you can't be bothered to setup infrastructure to manage building them on a project of yours that has C exts does not make them useless.

[ncoghlan]

Mike, wheels already solve a problem - there are projects that do have their own build farms (or a sufficiently large number of volunteers involved), and wheels allow those projects to take the burden of compilation off end users by providing pre-built binaries. For pure Python projects, it also makes it easy to speed up installation for end users, as installing from a wheel file is much faster than installing from source.

The challenge is for projects in the middle, where they do have C extensions, but don't have the capacity to provide pre-built wheel files for all supported versions on both Mac OS X and Windows. Since pip will fall back to installing from source if there's no appropriate wheel file available, it's entirely possible to create platform specific wheels for one or two popular OS and version combinations where making things easy for new users is particularly desirable (such as 2.7 and the latest 3.x on Windows), but that's still additional work that volunteers may not have the time or interest to pursue.

In those cases, it would be good if pythonwheels.com could offer a mechanism for the maintainers of affected projects to explain the problem, rather than having the site effectively label them as "bad". Reviewing the site, I also realised it inaccurately treats platform and architecture specific builds as inferior to pure Python packages: this is wrong, as the main purpose of wheels is to distribute pre-built binaries (while they do speed things up a bit in the pure Python case, that's a minor benefit compared to being able to skip the compilation step). Those projects should be green, not orange, as they're doing the right thing. That would free up orange to cover the cases where the maintainers have provided an explanation for why wheels aren't going to be available any time soon (and that's usually going to be "waiting on the availability of a public build farm for wheel archives").

A public build farm is a long term plan because it doesn't make sense to have everyone that wants to provide wheel files having to duplicate that infrastructure. However, it's not the only way to create them, publishing platform appropriate wheels does dramatically improve the experience for new users, and it's going to be quite some time before a public build farm is a feasible project for anyone to tackle - however, pythonwheels.com could provide a useful service in gauging interest in such a feature.

Note also that requests for behavioural changes in the packaging tools themselves should be directed to distutils-sig and the Python Packaging Authority (e.g. via https://github.com/pypa/packaging-problems) - the pythonwheels site is an unaffiliated project that just monitors publicly available package metadata.

[dstufft]

@ncoghlan Yea I think I mentioned that awhile back when pythonwheels.com was being created. That orange for platform specific Wheels didn't make much sense. It might make sense to make it orange if they don't have Windows wheels or something but in general making it look bad if you have platform specific wheels is wrong.

[zzzeek]

OK, we've established a. mike rants like a fool (we knew that) b. pythonwheels.com is a little misleading. So then also my original question: 1. what does "universal=1" mean, 2. if it means, "this is a pure python package", then why the heck does "python setup.py bdist_wheel" package .so files in the wheel file if universal=1? if those two conditions together make no sense, why should it proceed, or alternatively, if "universal=1" + "you have C extensions" does make sense, what does that mean ?

[dstufft]

universal=1 means that this wheel will work on all versions of Python. Generally it's only applicable to pure python wheels but it could also be applicable I suppose if a project has multiple .so's for different platforms inside it's Wheel and selects at runtime when importing which .so to actually import. But yes generally it's only useful for pure python.

[dstufft]

Also see #2 for the orange issue.

[dstufft]

Basically the way Wheels work, is the filename has a number of selectors/compatability tags in them. Pip tries to find the filename with the most specific set of selectors that still matches the current Python. If it can't find one it will fall back to a plain old source install. The universal flag just tells bdist_wheel to use py2.py3 as the python version selector instead of something like py27.

Under the cover Wheels are just zip files which when installed are essentially just unzipped and moved into the proper place. There is no support to compile things or otherwise modify the output. Essentially Wheel can be installed mostly with unzip and mv.

One great thing that can be done is to upload Windows and maybe OSX wheels for things that require compilation because compiler toolchains on those platforms, especially Windows, can be more complex to set up and a lot of users don't have them.

[zzzeek]

yes my original tweets regarding "what is the point?" are certainly more emotional than factual and of course being able to put a platform-specific build up is useful. However I do not see the point of a wheel file for a project that has no C extensions. I also think the "has multiple .so's present and is conceivably "universal"" idea you're getting at is nonexistent enough that there should at least be some kind of warning/ "are you sure you really want this mac-OSX .so to be "universal" and maybe instead you have no idea what you're doing and need to be alerted to that" type of message. A wheel that has .so's for multiple platforms is still not "universal". First of all, there are unlimited unix variants so no amount of .so files can ever be considered "universal", and secondly you'd at least need to see .dlls in that file as well for any semblence of "universal".

[dstufft]

Sure, there probably should be a warning and maybe even a flag to force it in that case. That would belong over on https://bitbucket.org/dholth/wheel. Sorry I didn't mean to sound like I was saying that was a reasonable use case. I was mostly spitballing in a "well here's one situation where maybe it would be useful" sort of way.

As far as the point for a pure python project. Wheels install faster, a lot faster. They also have all static metadata which means tooling can introspect them for things like dependency information without executing a setup.py. Installing a sdist involves several subprocess calls to Python each of which has a cost to starting up. Installing a Wheel is unzip and shutil.move (a long with a little bit of extra stuff for shebang rewriting and such). In the long run we'll get an sdist format that also has static metadata but there will always be additional work that has to be done in order to install an sdist, even a pure Python one, that just doesn't exist in a Wheel.

[dstufft]

Oh forgot to mention too, that the "cost" of making a pure Python Wheel is very low so there's little reason not to add one for a pure python project.

[zzzeek]

OK great, that is a reason then, I understand. So I think basically pythonwheels.com is a completely confusing and misleading site and someone should put up a site that lays the actual reasons out:

1. a wheel file is a useful, fast installing package for pure python or native C-code packages.
2. if your project uses C extensions, theoretically there'd be an individual wheel file for every possible architecture with pre-built .so or .dlls inside of it.
3. if your project does not use C extensions, wheels are still cool, because they are fast and have cool metadata and stuff.

4. to make a universal wheel file for a Python project:

   a. The Python code must run in Python 2 or 3 with no changes, including no 2to3, etc. b. The Python code must not have any C extensions. c. put the universal=1 flag in setup.cfg, here's how to build, etc. d. The system currently does not warn you if your project doesn't actually run 2/3 compat or in pure python.

5. Python projects that have optional C extensions that are generally desirable should probably not put up universal wheels, as their presence on Pypi will mean that pip will favor the non-C-building wheel over the source build.

[ncoghlan]

Yep, that summarises the situation well - I think Mike's last comment should be used as the basis for clarifying the text on pythonwheels.com itself, rather than it needing it to be done in a new site.

I've also filed issue #30, suggesting some changes to the current colour scheme (mostly avoiding the use of red for projects that don't publish wheel files yet).

[vsajip]

One more thing to mention - wheels built using bdist_wheel can have non-standard metadata (i.e. not conforming to PEP 426) in them, which means that other wheel implementations (such as distlib) can't process them. See this issue on the wheel project. This currently affects the official setuptools wheel, but I suppose other wheels could also have this problem.

We're supposed to be getting away from implementation-defined to PEP standards-compliant behaviour, so while pythonwheels.com ticks the good-intentions box, promoting bdist_wheel in its current state to build lots of wheels might be counter-productive.

[ncoghlan]

pydist.json is not part of the wheel 1.0 format, and PEP 426 is not an accepted specification. All currently defined aspects of the wheel format are in PEPs 425 and 427. Everything else is currently implementation dependent, and will remain so until PEPs 426, 440 and 459 are accepted.

[vsajip]

Right, but then perhaps pydist.json should be left out of wheels built by bdist_wheel altogether. Otherwise, if pythonwheels.com achieves its goal, there would be a lot of wheels on PyPI containing non-conforming pydist.json files, which would need to be rebuilt when PEP 426 gets accepted. Is that a desirable outcome?

[ncoghlan]

Vinay, I'll reply over on the bdist_wheel tracker - this really isn't @meshy's problem to worry about :)

[vsajip]

Agreed, though I put it here because I thought the caveat ought to be mentioned on pythonwheels.com.

[zzzeek]

thanks @ncoghlan and @dstufft for turning an alcoholic rant into productive changes!

[LilyFoote]

@zzzeek @ncoghlan @dstufft I've started trying to improve the pythonwheels documentation of some of theses issues in #33. I would appreciate some code review/fact checking.

[meshy]

Hey everyone! Thanks for the insight into the future plans for packaging! (Build farm -- exciting!)

Note also that requests for behavioural changes in the packaging tools themselves should be directed to distutils-sig and the Python Packaging Authority (e.g. via https://github.com/pypa/packaging-problems) - the pythonwheels site is an unaffiliated project that just monitors publicly available package metadata.

@ncoghlan Thanks for clarifying that, it is evidently a little unclear on pythonwheels.com, and I will try to see that something is done to make it clearer.

I get the impression that all the things that need to be actioned here are being addressed by the ever vigilant @Ian-Foote in #33, so I'm going to close this issue.

Thank you all for your participation. :)

[meshy]

PS. @zzzeek I'm sorry to hear about the C-extensions in SQLAlchemy.