search

pypa / twine

Utilities for interacting with PyPI
https://twine.readthedocs.io/
Apache License 2.0
1.61k stars 308 forks source link

keyring as optional dependency? #1008

Closed ax3l closed 1 year ago

ax3l commented 1 year ago

Your Environment

1) Your operating system:

Ubuntu 20.04 on PPC64le

2) Version of python you are running:

3.8 - 3.11

3) How did you install twine? Did you use your operating system's package manager or pip or something else?

pip

4) Version of twine you have installed (include complete output of):

latest

5) Which package repository are you targeting?

PyPI.org

The Issue

I am trying to upload my new ppc64le wheels with twine from Travis-CI (OSS-free partner queue sponsored by IBM). It looks like installing twine pulls cryptography through keyring as an unconditional dependency.

That install fails because on ppc64le there is no wheel for cryptography and then it falls back to a from-source build (Cryptography requires Rust >=1.41.0 which is hard to get on Ubuntu 20.04).

The Cryptography people (cc @reaperhulk) pointed me here, because they think the problem can be solved by making keyring an optional dependency of twine.

Steps to Reproduce

See: https://github.com/pyca/cryptography/issues/6086

sigmavirus24 commented 1 year ago

Duplicate of #837

ax3l commented 1 year ago

That issue does not address the problem that twine cannot be deployed anymore to Linux ppc64le. What is your solution?

How am I supposed to upload wheels on/for ppc64le, a fully supported platform for pip and pypi?

sigmavirus24 commented 1 year ago

As I said here https://github.com/pypa/twine/issues/837#issuecomment-984583453 making keyring optional is not an option.

Many good CI platforms enable separate build stages and upload stages so you can build your artifacts on whatever environments you care about and then upload them all at the same time.

In other words, this should be fixable in your configuration of your Travis CI matrix such that the ppc64le platform builds the wheel for you and another step with an architecture supported by cryptography and cffi runs the upload with twine. I strongly suggest exploring this option.

Alternatively, I believe many of the new generation of packaging tools all to various degrees have their own upload integration for PyPI and could potentially serve you well as well. Especially if you're already using one.

Twine's one job has always been to make uploads as secure as we can. Keyring is a dependency that allows us to do just that and it's not an unreasonable dependency for what it does.

ax3l commented 1 year ago

Many good CI platforms enable separate build stages and upload stages so you can build your artifacts on whatever environments you care about and then upload them all at the same time.

In other words, this should be fixable in your configuration of your Travis CI matrix such that the ppc64le platform builds the wheel for you and another step with an architecture supported by cryptography and cffi runs the upload with twine. I strongly suggest exploring this option.

Interesting suggestion. Honest question: which CI platforms support architecture swaps between build stages? And which of those support something else than x86 and arm?

Update: ok, docs where not helpful but I found an issue documenting it (from a former colleague, thanks @flamefire :D) https://github.com/travis-ci/travis-ci/issues/10230

Alternatively, I believe many of the new generation of packaging tools all to various degrees have their own upload integration for PyPI and could potentially serve you well as well. Especially if you're already using one.

At least cibuildwheel also relies on twine: https://cibuildwheel.readthedocs.io/en/stable/deliver-to-pypi/

Which packaging tools are you referring to?

Twine's one job has always been to make uploads as secure as we can. Keyring is a dependency that allows us to do just that and it's not an unreasonable dependency for what it does.

cc @reaperhulk just FYI. Your suggestion in https://github.com/pyca/cryptography/issues/6086 is not workable for twine. If we can revisit this ticket to build cryptography on Ubuntu ppc64le workers that would help me a lot :)

sigmavirus24 commented 1 year ago

At least cibuildwheel also relies on twine:

They document that you should use twine. If you take a very short amount of time (it took me less than 30s) you'll see it doesn't depend on it or install it for you https://github.com/search?q=repo%3Apypa%2Fcibuildwheel%20twine&type=code

Which packaging tools are you referring to?

hatch, poetry, etc.

Interesting suggestion. Honest question: which CI platforms support architecture swaps between build stages? And which of those support something else than x86 and arm?

Travis used to but they completely failed to manage abuse on their platform and had to shutdown a bunch of features. Their documentation was never great anyway, so not surprised you can't find it there.