pip should support custom wheel platform tags.

OptiverTimAll commented 9 years ago

As I understand it, the Wheel format has the notion of a 'platform tag', such that wheels built for the same version of Python with the same ABI but different platforms (say, CPython 3.4 on Windows vs. Mac OS X) will not collide. Although PEP0425 defines platform tags linux_i386 and linux_x86_64, PyPI will not accept uploads of wheels for those platforms because they are not specific enough—a linux_x86_64 wheel built on, say, Debian 8 might not work on CentOS 7, and so forth.

This is a good plan, and I support it.

However, inside the company I work for, the problem is much less complex - instead of a myriad of Linuxish platforms, we only have a handful, and we would love to be able to define custom platform tags for our internal packages, so we can be sure we're deploying the correct wheels to the correct hosts.

To be specific, I want a workflow something like the following:

I'm using Pip 7.0, so I have a shared wheel cache.
My CI system notices that somebody has committed a change to my project, so it tells a member of the build-farm to run the tests.
The build host happens to be one of our legacy CentOS 6 machines, so when it populates a virtualenv to run the tests, it runs:
```
pip install --editable --platform-tag optiver_centos_6 ./source/
```
...and so when pip consults its wheel cache, it only considers wheels whose platform tag is optiver_centos_6 or any. Furthermore, if it builds fresh wheels and caches them as part of the installation process, they will be tagged with optiver_centos_6 if they contain C extensions (or whatever the usual logic is).
Most likely, CI system will also run the tests on CentOS 7, doing the same steps but with --platform-tag optiver_centos_7.
Later, when my project has passed all its tests and is ready for deployment, the build host should be able to gather a complete collection of wheels compatible with the given platform:
```
pip wheel --platform-tag optiver_centos_6 ./source/
```
As before, --platform-tag should govern which wheels are pulled from the cache, as well as tagging wheels that are freshly-built.

rbtcollins commented 9 years ago

I presume your CI has a shared filesystem, NFS or something, with a common home dir? I have the same thing on my laptop, via LXC containers with home mapped into the container; and I'm expecting the shared cache (that I wrote :)) to bite me soon in that setup. SO yes - I like this idea. I'd want @dstufft's input on the approach before implementing though.

dstufft commented 9 years ago

I have mixed feelings about defining your own platform tag, but I think that the right place to deal with the general question of how to handle Linux platforms (or platforms in general) is via distutils-sig/Wheel. Once we have some standard way of doing it (even if it's "define your own platform tag") we can expose that in pip as it makes sense to do so for whatever solution we end up with.

OptiverTimAll commented 9 years ago

If you'd rather I bring the matter up on distutils-sig, I'd be happy to do that, but I think Pip should support custom platform tags no matter what. PEP0425 says "...built distributions should have a file naming convention that includes enough information to decide whether or not a particular archive is compatible with a particular implementation" but I believe "compatibility" is a matter of opinion.

For example, I might want to ensure that my production deployments only include wheels compiled with -D_FORTIFY_SOURCE for security hardening. Wheels built with and without such a flag might be compatible in that they both pass a test suite, but they would not be compatible if you're looking at "likelihood to pass security penetration testing".

Alternatively, if I have a C extension written to speed up some task, I might want to ensure that a wheel built with -march=native is only ever deployed to hosts with exactly the same micro-architecture. A wheel built for a different micro-architecture might calculate the same answer, but it might run unacceptably slowly for my purposes. Or it might even crash, if the other micro-architecture supports some extra CPU instruction.

Even if two environments should be compatible, and distutils-sig gives them the same platform-tag, I might want to distinguish them just to avoid the risk that they might somehow be incompatible in some subtle way I haven't noticed yet.

It would be silly and unmanageable for pip to fill the platform-tag with every possible difference that could ever possibly affect anyone, especially since even the people that care probably only care about one or two of them, and it's a different one or two for each person. However, I think it's reasonable to say "if you have compatibility concerns that the official platform-tag system doesn't represent, you can make up your own platform-tags as fine-grained as you want as long as you keep them to yourself and don't upload them to PyPI".

dstufft commented 9 years ago

I think there are compelling cases for custom platform tags for sure, pip doesn't actually build wheels though- it just calls out to the wheel project to do that. So at a minimum the wheel project would need to support it but we could (unless we did something gross, like rename the file after it was created, but i'd rather not do that).

pfmoore commented 9 years ago

And of course the fact that this affects not just the pip project, but also the wheel project, is why it's best discussed on distutils-sig :-)

rbtcollins commented 9 years ago

I've raised it on distutils-sig

rbtcollins commented 9 years ago

And distutils-SIG says 'sounds good to us'

OptiverTimAll commented 9 years ago

Sweet!

I should note that setup.py bdist_wheel already supports --plat-name which I believe overrides the platform tag of created wheels, so the wheel project should already be covered.

dstufft commented 9 years ago

I don't think it works unless you specify a previously allowed one:

$ python setup.py bdist_wheel --plat-name foobar

...

Traceback (most recent call last):
  File "setup.py", line 307, in <module>
    **keywords_with_side_effects(sys.argv)
  File "/Users/dstufft/.pyenv/versions/3.4.3/lib/python3.4/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/dstufft/.pyenv/versions/3.4.3/lib/python3.4/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Users/dstufft/.pyenv/versions/3.4.3/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/dstufft/.pyenv/versions/3.4.3/lib/python3.4/site-packages/wheel/bdist_wheel.py", line 213, in run
    archive_basename = self.get_archive_basename()
  File "/Users/dstufft/.pyenv/versions/3.4.3/lib/python3.4/site-packages/wheel/bdist_wheel.py", line 161, in get_archive_basename
    impl_tag, abi_tag, plat_tag = self.get_tag()
  File "/Users/dstufft/.pyenv/versions/3.4.3/lib/python3.4/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
    assert tag == supported_tags[0]
AssertionError

rbtcollins commented 9 years ago

Ok, so we need to open an issue on wheel to make it possible to override (whether same option or a new one), then we can tackle it in pip.

OptiverTimAll commented 9 years ago

I've opened an issue on wheel: https://bitbucket.org/pypa/wheel/issue/144/bdist_wheel-should-support-custom-platform

OptiverTimAll commented 8 years ago

The issue for wheel has now been fixed.

natefoo commented 8 years ago

And now released in wheel 0.27.0.

blsmth commented 8 years ago

this was fixed in the wheel project... any plans to incorporate this into pip? currently I can build wheels with a custom platform-tag but have no way to easily install them with pip.

i know i could write some scripts that go through and try to find requirements via curl calls to my private pypi server, but pip install -r requirements.txt --plat-name centos7.1 would be much nicer 😄

dholth commented 8 years ago

It would make sense to have something like an additional_supported_tags in the pip config file...

cattz commented 7 years ago

If this is still not fixed in pip, how are you guys using the custom platform tags?

pradyunsg commented 7 years ago

Did this get fixed at some point with --platform or I'm missing something and there's stuff to be done here?

rudolphfroger commented 7 years ago

Custom platform specifier would be very welcome! Under FreeBSD the platform strings used by wheel are unusable (like 'freebsd_11_1_release_p1') and should probably be something like 'freebsd_11'. Supporting custom platform names would solve this.

analogue commented 6 years ago

fyi, pip-custom-platform

cattz commented 6 years ago

@analogue, thanks for the suggestion. pip-custom-platform is a nice alternative, but I think it would make a lot more sense to support this "natively" in pip, without the need of additional packages. Think, for example, using custom platform tags in an ansible playbook, to deploy your application

pradyunsg commented 6 years ago

I've labelled this issue as an "deferred till PR".

This label is essentially for indicating that further discussion related to this issue should be deferred until someone comes around to make a PR. This does not mean that the said PR would be accepted, though I personally am fine with going ahead with this feature.

pypa / pip

pip should support custom wheel platform tags. #2875