search

pyca / cryptography

cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.
https://cryptography.io
Other
6.65k stars 1.52k forks source link

Dependency on rust removes support for a number of platforms #5771

Closed mgorny closed 3 years ago

mgorny commented 3 years ago

I would like to report that the newly added dependency on Rust has made it impossible to package cryptography for a number of supported Gentoo architectures (and these are architectures where people actually use Python packages that depend on cryptography).

Please see Platform Support. Besides degrading a few of our targets to 'Tier 3', Rust completely does not support at least alpha, hppa, ia64, m68k, s390 (not -x). The cost of adjusting cryptography's C code to these platforms (even if we assumed they wouldn't work out of the box as they do so far) is much less than the cost of porting the whole Rust. You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

Right now it's a really hard struggle to even get Rust packaged for non-Tier 1 architectures (i.e. where upstream does not provide bootstrap binaries).

LindezaGrey commented 3 years ago

destroyed my CI/CD Pipeline this morning :laughing:

clanzett commented 3 years ago

It is also impossible to build on ANY alpine linux version/architecture. Please remove this dependency as soon as possible.

tiran commented 3 years ago

It is also impossible to build on ANY alpine linux version/architecture.

Update to most recent Alpine with Rust >= 1.45. Alpine latest has rust (1.47.0-r2).

Please remove this dependency as soon as possible.

Not going to happen. Rust dependency will stay to replace C code with an actual safe language.

You can disable Rust integration in 3.4.x (see FAQ). Starting with 3.5 cryptography will have mandatory Rust code.

clanzett commented 3 years ago

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah... But blowing up systems with unnecessary software packages is a more secure way, that's for sure... (my 2 ¢)

mgorny commented 3 years ago

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

tiran commented 3 years ago

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

But blowing up systems with unnecessary software packages is a more secure way, that's for sure... (my 2 ¢)

The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.

silverfisk commented 3 years ago

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

clanzett commented 3 years ago

Memory safety is of course a compelling reason, but you should also keep in mind that may developers out there are using your package, which is great by the way, and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on. A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

clanzett commented 3 years ago

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

Smirl commented 3 years ago

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time. From the changelog:

BACKWARDS INCOMPATIBLE: Support for Python 2 has been removed. We now ship manylinux2014 wheels and no longer ship manylinux1 wheels. Users should upgrade to the latest pip to ensure this doesn’t cause issues downloading wheels on their platform. cryptography now incorporates Rust code. Users building cryptography themselves will need to have the Rust toolchain installed. Users who use an officially produced wheel will not need to make any changes. The minimum supported Rust version is 1.45.0. cryptography now has PEP 484 type hints on nearly all of of its public APIs. Users can begin using them to type check their code with mypy.

This should probably be a 4.0 release it would cause a lot less pain for the community. I suppose many have cryptography>=3.0,<4 in setup.py or requirements.txt.

Update I was unaware that you don't use semantic versioning. I have found in the docs https://cryptography.io/en/latest/api-stability.html#versioning. However it was a little confusing given the style. As many of our dependencies require this we will try not to pin a version and install rust/use manywheel packages.

tiran commented 3 years ago

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

PS: I've been preparing Fedora packaging for almost three months. My Fedora Rawhide packages of python-cryptography 3.4 landed about 12 hours after the release -- and I had to jump through several extra hoops because Fedora does not permit vendoring of Rust crates.

mgorny commented 3 years ago

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

tiran commented 3 years ago

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

public commented 3 years ago

@tiran

I wonder if there's a way to produce a more helpful error message, perhaps with a link to some documentation about the new required build deps. The current message most people will hit is

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed.
      =============================DEBUG ASSISTANCE=============================

Which does sort of attempt to guide people the right way but could be more specific for common cases like Alpine.

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

mgorny commented 3 years ago

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

That's a very roundabout way of showing the middle finger to users of 'other hardware platforms' that are not supported by Rust (or LLVM) at all.

fourstepper commented 3 years ago

CI testing went down the drain with this in GitLab CI with docker:stable-dind, as it's based off of Alpine. I can imagine this happening all over the place.

tiran commented 3 years ago

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

The documentation for Alpine builds has been improved https://github.com/pyca/cryptography/pull/5763 already. I have a pending PR for Fedora, CentOS, and Debian at https://github.com/pyca/cryptography/pull/5764 . If you have any suggestions for better build instructions, please help by opening a PR.

thesamesam commented 3 years ago

For myself, I'd like to note that a heads-up in the CHANGELOG / Release Notes (I may have missed it) would've helped: 1) when considering the move ("we're considering doing this, please give feedback") 2) when it was decided it should actually happen ("release X will optionally contain Rust, release Y will have mandatory Rust")

I don't see any mention before the release in the CHANGELOG or in the emails to python-announce-list.

That's the one thing which distro maintainers or packagers really should be reading on every new release. Including it as a warning in previous releases would've helped us out here to both object/raise concerns and begin preparations.

cdesousa commented 3 years ago

To be fair the developers did ask for feedback (in a not very inviting way):

We are interested in feedback in things we can do to make this a smooth process, particularly from folks who are re-distributing cryptography. Please note: feedback of the form “don’t use Rust” is not productive and will be disregarded

I highly suspect there will be many more complaints throughout the day about stuff breaking. I had stuff breaking on amazon linux and was confused as to why pip install ansible was asking for rust, seems that one is tagged to latest cryptography in requirements.txt (which is of course not directly the fault of the devs here).

MrMino commented 3 years ago

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about first impressions :smile:. That's a good reminder to update the alpine images though.

Hands up: who here watches every update of every transitive dependency in each app/environment they have to maintain?

BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?

clanzett commented 3 years ago

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries. I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about reminders to update to the newest alpine 😄.

I totally agree. I really think we did not even scratch the tip of the iceberg. There might be some "interesting" problems ahead.

And don't get me wrong @tiran, I also agree with you regarding your security concerns, but this switch could have really done some serious damage. We will see...

xgqt commented 3 years ago

Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

other hardware platforms

lol

ambv commented 3 years ago

It sounds like y'all are getting an unexpected lesson about the flip-side of open source software. I would like to push back on some of the entitlement I'm sensing reading in this thread.

Before I begin, I'd like to remind you that security is a numbers game. If the cryptography maintainers can help 90% of their users by switching to a modern memory-safe language, then they'd be irresponsible holding back just because among the remaining 10% there exist fringe platforms which can't even run a Rust compiler.

You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer is worth more than the time of cryptography maintainers. They are volunteers as well, y' know. You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.

In fact, your only real recourse long-term is exactly to band up volunteers to make Rust work on your platform. Or pay some hired guns to do the initial work for you. After all, you will be seeing more and more projects depending on this programming language as time goes on. Yelling at creators of those projects for choosing a language that is inconvenient for you is both ineffective and laughable.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

I invite you to do that. Please put your money and time where your mouth is. Report back in a year's time how it went.

xgqt commented 3 years ago

Let's depend on Rust because everything does! Let's depend on glibc because everything does! Let's depend on GNU coreutils because everything does! Let's depend on systemd because everything does! Let's depend on Linux because everything does!

Afternote: should be "HARD depend" - without the ability to toggle building with the given component.

manuelmarquezs commented 3 years ago

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. The build of the cryptography package that used to take a few seconds now takes minutes.

We have updated alpine, installed all the dependencies listed in the docs: sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo and we are using Rust>1.45.0.

Any idea on why the compilation with Rust is that slow?? How could we speed it up??

Building wheels for collected packages: cryptography
  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): finished with status 'done'
wvxvw commented 3 years ago

It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.

cryptography is a transitive dependency in our code: through jira, which requires oauthlib. Of course, in part, the blame is on pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version. But you, as a maintainer, also did a very bad job at signaling this to pip.

If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.

I'd be totally fine with there being a cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.

felixfontein commented 3 years ago

@wvxvw creating a new package would actually make the situation a lot worse. Except if the Python module name is also renamed, which would require everyone to adjust there code. Which would also be worse IMO.

tiran commented 3 years ago

It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.

Ubuntu Bionic LTS has recent Rust and Cargo version. pip is the problem here. Ubuntu 18.04 comes with an old, unsupported pip version. I recommend that you install a newer pip version in a virtual env. It might also be worth the trouble to file an update request with your vendor.

# echo $(source /etc/os-release; echo $VERSION)
18.04.5 LTS (Bionic Beaver)
# apt update
# apt install -y rustc python3-pip
# rustc -V
rustc 1.47.0
# pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
MrMino commented 3 years ago

@ambv I'd agree with everything you've said, if it wasn't for this line:

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

With all due respect, it seems to me like this is a strawman. We're not talking about a security update here, are we?

I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.

Currently the lockfile in one of repositories I maintain shows 354 dependencies. The main dependency list includes around 50. The app that is broken for me is a staging environment for a distributed hardware testing platform. This is as far from security concerns as you could get, there's no way I'm able to inspect every single transitive dependency for updates, and there's even less probability that management permits me to spend my time on it.

Please have some empathy. I'm really not blaming any cryptography maintainer, I don't have the time and energy to - I'm fighting an NP-complete problem and this update is making my life significantly harder currently.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer

We have been able to fix our alpine Pipelines [...] but they are now extremely slow. We have gone from 30s to 4min

It's really not just "some other platforms". Ubuntu 18 is also affected...

Also, now It's not just about Gentoo, is it? Don't you think the 90% figure is a bit exaggerated?

wvxvw commented 3 years ago

@felixfontein absolutely false... I'd be able to live forever with cryptography instead of cryptography2. And, in case I really wanted the update, I'd be able to update. pip integration is the major problem here, as I just wrote. There's no good way to tell pip that new version of the package is incompatible, beside renaming the package. I believe that the change made to cryptography warrants such a move.

@tiran Sorry, not going to happen. I'm neither updating pip, nor am I installing Rust in those images. Our product is also written in Rust, and your dependency on Rust is completely unnecessary for us to consider in our CI. pip is the version it is for a lot of reasons too.

For me, a solution that would involve embedding the entire jira package with its transitive dependencies in our own deployment would be easier than trying to deal with your demands to our CI environment...

felixfontein commented 3 years ago

@wvxvw have you ever dealt with cases where pip packages were renamed, but the corresponding Python modules weren't? If you didn't, believe me: it's not fun.

MrNaif2018 commented 3 years ago

@tiran Just my two cents here. I would say, that any improvement that reduces the risk of security vulnerabilities is a good improvement. If rust really improves the memory management and helps avoids some future vulnerabilities-go for it. But there are only two things that concern me:

Of course, probably nothing can be done right now after the release is done. Or maybe could it be implemented that way?

I am also using cryptography as a non-direct dependency in docker image builds (alpine linux, via electrum)

wvxvw commented 3 years ago

@felixfontein correct me if I'm wrong, concurrent.futures and pathlib would be the examples, maybe urllib too.

Of course, it's not nice, but it's a problem that is fixable entirely on my side. I cannot fix the dependency on cryptography for others though, not on my side at least, w/o completely taking over another package.

felixfontein commented 3 years ago

@wvxvw for pathlib / pathlib2, the name of the Python module changed (from pathlib to pathlib2). In that case, a rename makes sense, because the semantics change. (I haven't looked at the others. I guess they will have similar reasons.) In any cas,e that's not the case for cryptography, using the library does not change (except possible removals of deprecated features). Just renaming everything because a new dependency was added is pretty extreme and very unfriendly to most users of this library.

Different topic: doesn't a simple constraints file with cryptography < 3.4 help in your case?

alex commented 3 years ago

👋 good morning everybody. This ticket contains a lot of different complaints about the 3.4 release, which incorporates Rust. First: I'm sorry this transition has not been as smooth as we'd all like.

Some context: For several years, @reaperhulk and I have been doing research and publishing on the security threat posed by memory unsafe languages like C. Just last week I spoke at Enigma on this. We believe long term adoption of memory safe languages is nothing short of a moral imperative.

I'm not going to rehash everything that's been said in this thread, but to summarize my views: We made substantial efforts to communicate this change in advance, it's possible to disable this change with a simple environment variable (for this release only), we invested a lot of our time and energy is improving the Rust integration with python packaging (including contributing PEP384 support to pyo3).

And finally: No, I don't care that Rust is not well supported on HP-UX, AIX, and similar. You'll notice I'm being cavalier and not at all empathetic on this particular point. That's because these are incredibly minority platforms, and we (speaking more broadly) can't hold up security improvements for platforms stuck in the past.

On a philosophical note, I'm surprised to discover that so few users of Python packaging employ lockfiles/exact version pinning.

With all that said, let's get constructive!

I think there's a few categories of issues going on here, let's talk through them -- I'll sketch out my views and would appreciate feedback:

1) There's lots of users who could be consuming wheels (in which case they have no need for Rust to be installed), but for the fact that their pip is out of date. Here, I'm not sure there's anything to be done but encourage people to update their pips -- we already do this in every piece of documentation we have.

2) Users with pips that predate PEP517 get an ImportError. This is another case where updating pip resolves it, and we've made a change to have this print out additional debug information.

3) Users with older versions of rust installed experience nasty compilation errors. Here I think we can a) improve our docs to clearly indicate the version of rust required, b) more clearly suggest installing rust with the docs from upstream, c) think about how we might implement the version check directly and thereby give a clear error message (https://github.com/rust-lang/rust/issues/65262#issuecomment-763765009 would be best, but it's not yet stable)

4) It's clear that a lot of alpine users would benefit from having wheels. Someone (not me!) ought to do the work to write a PEP for an alpine build tag, then we'll ship wheels for it and reduce the burden on all those users.

5) It's clear that, despite our best efforts to communicate this, folks were taken by surprise. Where would you have liked us to post this circa 6 months ago?

webknjaz commented 3 years ago

@clanzett

and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on

I see how this may make you feel frustrated I don't think anybody is forcing you to do anything. It's completely your choice like you chose to use the library out of your own free will (instead of implementing your own, I guess). Blaming things on others is rather destructive thing to do. Besides, did you pay the maintainers for any of the great work they've been doing all these years? Did you pay them for not introducing a safer technology? This comes off as thankless.

@Smirl

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time.

Nobody promised semver and it's publicly documented at https://cryptography.io/en/latest/api-stability.html.

@MrMino

BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?

I think a combo of pinned envs + dependabot (or similar) + CI can catch this just fine. Pinned envs are reproducible, and any incompatibilities are clearly visible in automated bot bumps when they explode in CI w/o automatically breaking the main branch.

@manuelmarquezs

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. [...]

Any idea on why the compilation with Rust is that slow?? How could we speed it up??

How about:

  1. Cache base docker layers/images (as in all the static deps are built ones and so install+compile is not invoked in runtime)
  2. Cache pip wheels — pip builds wheels out of sdists when they are installed for the first time, any subsequent installs that select the same version will pick up those wheels and just unpack them (platform-specific wheels are zip archives with pre-compiled C-extensions)

@wvxvw

[...] Of course, in part, the blame is on pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version.

Hey, let's not go into assigning blame here. Plus there's absolutely no fault of pip here — it resolves the deps correctly per declared metadata. Besides, only people involved in the development can dictate what should be done and how, consumers who just come in to do the blame and expect to benefit from free labor are in no position to demand how the volunteer efforts are being spent. The maintainers don't owe you anything — by using this software you agreed with its license that says that it's provided "as is" (the license files are in the repo, I encourage you to read them). Also, you are free to fork the project and set your own rules there (for as long as they comply with the licenses, of course).

But you, as a maintainer, also did a very bad job at signaling this to pip.

This is not true, the maintainers did a good job within what the current state of the Python packaging allows. I do agree, though, that the packaging ecosystem could be improved. But again, it's run by volunteers meaning that you could influence it by making a volunteer effort. It would be a good idea to even just participate in discussions around improving the distribution package metadata and contribute your use-cases and ideas. Those discussions can be found at https://discuss.python.org/c/packaging/14.

Some topics for improvements are tags for Alpine, as @alex rightfully mentioned, and less widespread things like BSD and others. I believe this would require somebody working on a PEP similar to ones introducing manylinux.

If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.

I'd be totally fine with there being a cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.

It would be a good suggestion at the time when the maintainers were collecting the feedback, not post-mortem.

@MrMino

I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.

It is covered, see: https://cryptography.io/en/latest/api-stability.html. In particular, it says:

X.Y is a decimal number that is incremented for potentially-backwards-incompatible releases.

@wvxvw

Our product is also written in Rust, and your dependency on Rust is completely unnecessary for us to consider in our CI. pip is the version it is for a lot of reasons too.

You seem to expect deprecated software to be supported virtually forever. I believe it's absolutely possible, you'll just need to pay somebody to perform such a maintanence.

@felixfontein correct me if I'm wrong, concurrent.futures and pathlib would be the examples, maybe urllib too.

I believe Felix is talking about https://github.com/pypa/pip/issues/8509 — TL;DR because pip doesn't yet have a concept of transactions, files claimed by two different packages get wiped out on upgrade.

mcejp commented 3 years ago

Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Presumably I'm doing something wrong, but this hit me on Debian 10

webknjaz commented 3 years ago

@mcejp

Presumably I'm doing something wrong, but this hit me on Debian 10

Sounds like you've hit an old version of pip as mentioned before. Be aware that deb-like OSs have policies in place that hugely change the whole Python experience in those envs: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e.

MrMino commented 3 years ago

@alex I've created https://discuss.python.org/t/wheels-for-musl-alpine/7084 for Alpine.

Where would you have liked us to post this circa 6 months ago?

There's no one place that could've fulfilled this purpose. To me the issue is that we as a community don't have standardized channels for such communication. I think this issue is bigger than the project in question.

@webknjaz

It is covered, see: https://cryptography.io/en/latest/api-stability.html.

Yup. And that scheme failed us today, when lots of maintainers realized that some of the dependencies they use are made with total disregard for this. People assume it's semantic. We either start acknowledging that fact, or we'll be introducing a major breakage in the ecosystem every other quarter.

Assumptions are the mother of all f*ck ups

webknjaz commented 3 years ago

Yup. And that scheme failed us today [...] Assumptions are the mother of all f*ck ups

TBH there are a lot of packages in Python ecosystem that use this scheme starting with ansible-core, ending all the packages that use CalVer, for example. Going further, among OSs that many in this thread use, there's Ubuntu that does CalVer too. So it's not like there's an assumption in the community that every project would use SemVer automatically. Reading the docs about the versioning expectations is on those who specify the dependencies in their envs, many libs don't have this properly documented but this one does and it is very clear about what to expect...

There's no one place that could've fulfilled this purpose. To me the issue is that we as a community don't have standardized channels for such communication. I think this issue is bigger than the project in question.

The docs at https://cryptography.io/en/latest/community.html point at an ML and an IRC channel. This should be enough honestly. More places make it harder to track things... OTOH I'd probably encourage the maintainers to enable Discussions in the repo settings as this provides a great platform for Q&A style posts.

MrMino commented 3 years ago

So it's not like there's an assumption in the community that every project would use SemVer automatically

I'm not talking about end-user apps and huge behemoths such as Ubuntu. I'm talking packaged APIs. I haven't checked, but I think that if I do, it will turn out that the vast majority of packages on PyPI use SemVer. Or any scheme that uses "if the first digit changes, you'll have to open your IDE, sorry".

More places make it harder to track things...

No, not forums to discuss it - a functional scheme (e.g. in PyPI), that would allow a maintainer to notify their users about a breaking change in advance. We all do this using a weird amalgamation of emails, forum posts, pinned issues and warning messages. Why?

The signal to noise ratio for this is so small that it becomes tedious to stay alert.

I don't want to subscribe to every possible message on the topic there is, I'd like to have something that gives me a clear signal that says "dude, if you don't fix your crap in 1 week, it will end up in flames".

alex commented 3 years ago

FWIW, our version/API compatibility scheme comes from Django.

A PyPI feature to let us pro-actively notify our users seems like a good idea, I'd encourage folks to discuss that on the PyPI issue tracker!

clanzett commented 3 years ago

@webknjaz :

I did never blame anybody. I just said, that this change is forcing developers to adapt their CID/CDs a.s.o. and asked to take back this change. Nothing else....

The problem is, that this great library is used as a dependency in a lot of other packages and I guess many users are not aware of that. So just using another library won't fix anything - at least not for me - because that would mean that I also would need to change many others.

In fact I am very thankful for all the work, but I think open source does not mean, that I can't speak about my concerns! So please read my comments or try to understand them instead of accusing my of being not thankful!!

MrMino commented 3 years ago

@alex

FWIW, our version/API compatibility scheme comes from Django.

Which, honestly, might not be a good idea. Django assumes that it runs your code (i.e. inversion of control), not the other way around, so breaking changes are unlikely to have that much of a fallout there.

Simply put, Django, Ubuntu, Chrome, Pip, & others that use non-semantic versioning are almost never a transitive dependency. That's why using CalVer there doesn't come with a high cost for people that use it. In the case of Cryptography however...

webknjaz commented 3 years ago

@MrMino

No, not forums to discuss it - a functional scheme (e.g. in PyPI), that would allow a maintainer to notify their users about a breaking change in advance.

This is something that should go to https://github.com/pypa/warehouse or https://discuss.python.org to see if people would want to discuss implementing it.

We all do this using a weird amalgamation of emails, forum posts, pinned issues and warning messages. Why?

The signal to noise ratio for this is so small that it becomes tedious to stay alert.

I don't want to subscribe to every possible message on the topic there is, I'd like to have something that gives me a clear signal that says "dude, if you don't fix your crap in 1 week, it will end up in flames".

I think that the deprecation warnings serve this purpose rather well. But in this case, instead of a deprecation warning that would be happening in runtime, there was a build-time error that I'd consider a sort of a warning too: there's no real functional Rust code on the main branch right now — it only has a no-op with the main purpose of battle-testing the build toolchain setup at https://github.com/pyca/cryptography/blob/master/src/rust/src/lib.rs. I believe that this can be considered a "soft bump" of the build deps, it's easy to disable for one version with an env var, as described in the FAQ. It seems to be way smoother than merging lots of Rust and using it right away. The mechanism with the env vars is almost the same as with any previous deprecation.

@clanzett

In fact I am very thankful for all the work, but I think open source does not mean, that I can't speak about my concerns! So please read my comments or try to understand them instead of accusing my of being not thankful!!

It's good that you clarify this now: the problem is that in the text medium many language constructs can carry some sort of emotion and we need to care not only about what we write but also how we do it because it's somebody else who is going to experience the reading and can perceive the same words differently. Especially with the surrounding context saying that they "did a very bad job", or "in part, the blame is on pip" — this explicitly says "blame", I didn't interpret it in an arbitrary, unless there's another meaning of the word that I don't know...

sabueso commented 3 years ago

FIRE IN THE HOLE

MrMino commented 3 years ago

@webknjaz

I think that the deprecation warnings serve this purpose rather well.

You could've put a large ASCII art of a kitty in the warning message and I still wouldn't notice. The reason we have automated pipelines is so that we don't have to look at them until they give us a signal.

alex commented 3 years ago

Don't think I won't start including giant ASCII doggos...

I agree wholeheartedly about the merits of automation, and it's clear there's gaps here where there's no infrastructure available for some of that.

That said, I think the folks most fully embracing automation (pinned dependencies + dependabot for updates) would have experienced this relatively smoothly.

ambv commented 3 years ago

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

With all due respect, it seems to me like this is a strawman. We're not talking about a security update here, are we?

@MrMino, the OP here is a Gentoo package maintainer, not an end user. You'd expect repackaging security-related libraries is done with more care than a simple fire-and-forget script in CI.

I think (though I'm not sure now), that we have a reasonable expectation in this industry, that such changes will be covered by the versioning scheme.

While this particular project's versioning scheme is different, I agree that a 4.0 version number would signal the dependency change in a clearer way. Maybe they'll make 3.5 into a 4.0 because there you won't have a magic "get out of jail free" environment variable.

Ultimately though the issue we're commenting on would still be created by someone and we would still be having the same discussion. The developers did communicate their change through their available channels and it was missed/ignored. And most people have this dependency as a transitive dependency.

Currently the lockfile in one of repositories I maintain shows 354 dependencies. The main dependency list includes around 50. The app that is broken for me is a staging environment for a distributed hardware testing platform. This is as far from security concerns as you could get, there's no way I'm able to inspect every single transitive dependency for updates, and there's even less probability that management permits me to spend my time on it.

If Nokia's management is fine running random code from the Internet on your infrastructure then that's your call but ultimately large organizations ought to do some sort of dependency gatekeeping. I can tell you this from first-hand experience as this used to be one of my responsibilities. You can find other FAANG employees telling you the same.

In any case guilting unpaid volunteers by mentioning their actions will inconvenience your paid job isn't a good view. If the maintainers are unable to maintain an LTS version with C code alongside the current version based on Rust, nothing you can say will change that.

Also, now it's not just about Gentoo, is it? Don't you think the 90% figure is a bit exaggerated?

Note that Rust is available on Gentoo, just not on fringe architectures like alpha, hppa, ia64, m68k, and s390. None of those are even actively supported by CPython. s390x and ppc64le are "best effort" and at least we have stable buildbots for them.

90% was obviously a rhetorical figure but we can roughly estimate it for instance by looking at the PyPI download stats for the most popular dependency for Python. On February 6th, it has seen 3.17 million downloads on Linux. That's 96% of the downloads. So Alex's claim that non-Linux platforms are "incredibly minority platforms" is well supported by those numbers, especially if you take into account that another 2.5% is Windows.

We can also look at Python version proportions and we'll see that on that same day 60% of downloads were from Pythons 3.7 - 3.9. That means they're newer than Ubuntu 18.4 LTS which ships with Python 3.6, which also has been established can be fixed by upgrading pip. In fact, 19.9% downloads were from EOL versions of Python, including Python 2.7 - 3.5 which are irrelevant in this discussion because cryptography 3.4 does not run on any of them. So excluding those from the proportions, roughly 75% of global downloads are from Linux running Python 3.7+.

It would be cool to see those downloads per distro, per CPU architecture, or per pip version but it is what it is. So yeah, somewhere between 75% through 95%. We'll see this number move up as Debian 11 gets released this year, and so on. For others, upgrading pip or using rustup is a one-time change that the user needs to make to carry on. This includes Gentoo, Ubuntu 18.4 LTS, and Alpine Linux. Unless you're on m68k...

Finally, this is somewhat off-topic but Alpine Linux itself is a weird choice for a Python container given that it's slow and CPython does not run any stable buildbots on either Alpine or any other musl-based distro.

tiran commented 3 years ago

Note that Rust is available on Gentoo, just not on fringe architectures like alpha, hppa, ia64, m68k, and s390. None of those are even actively supported by CPython. s390x and ppc64le are "best effort" and at least we have stable buildbots for them.

Nit pick: s390 and s390x are different platforms. According to Wikipedia, s390 was a mainframe system released in 1990 and discontinued in 1998.

Cryptography's Rust extension builds fine on s390x. https://koji.fedoraproject.org/koji/buildinfo?buildID=1704720