Future versions of cryptography will require rust

Jc2k commented 3 years ago

Context

I'm raising this issue because of a tweet i saw [1] and I wanted to make sure that the HA team were aware in good time as it could require some changes to the release pipeline.

Previously we have rejected dependencies that relied on Rust due to complications in installing for the community. IIRC, there was quite a performant JSON implementation that fell into this category? E.g. https://github.com/home-assistant/core/pull/32185. As far as I know the situation has not changed since then (apologies if it has).

Unfortunately the cryptography library is now starting to rely on Rust. (Unfortunately for us i mean - I think it's great for cryptography). I believe this is a pretty core dependency in HA, and many of our dependencies depend on it too.

Cryptography 3.4 will have an optional rust extension. But cryptography 3.5 will require a rust toolchain to build from source as the extension will become mandatory.

In contexts where a wheel is available nothing should change. However I believe our Docker images are not glibc based and cannot use upstream wheels? The same situations applies to raspberry pi venv users too - it will complicate the install for them until piwheels ships a wheel for rusty cryptography.

[1] https://twitter.com/alex_gaynor/status/1341470025467740163 https://mail.python.org/pipermail/cryptography-dev/2020-December/000998.html https://github.com/pyca/cryptography/issues/5381

Proposal

Officially support python packages that use Rust extension modules to enable the continued use of the cryptography module going forwards.

I believe the approach would be to make a rust toolchain available when building wheels. There is some example GitHub actions YAML within the cryptography PR here. I don't know if any other parts of the build process would need to be adapted as well - does CI run on alpine? If not i believe it will use the upstream wheel and not be affected.

I don't know how we help e.g. Raspbian venv users, though.

Consequences

If we do not do this then we will be stuck on an old version of cryptography and eventually there will be a security vulnerability that forces our hand.

Removing cryptography from our stack will be hard. It's used directly or indirectly extensively. At least the following (based on pip dep tree in the HA docker image):

  - acme==1.9.0 [requires: cryptography>=1.2.3]
    - hass-nabucasa==0.39.0 [requires: acme==1.9.0]
  - adal==1.2.5 [requires: cryptography>=1.1.0]
    - msrestazure==0.6.4 [requires: adal>=0.6.0,<2.0.0]
      - azure-servicebus==0.50.3 [requires: msrestazure>=0.4.32,<2.0.0]
  - adb-shell==0.2.1 [requires: cryptography]
    - androidtv==0.0.56 [requires: adb-shell>=0.2.1]
  - aiohomekit==0.2.60 [requires: cryptography>=2.9.2]
  - apns2==0.3.0 [requires: cryptography>=1.7.2]
  - asyncssh==2.4.2 [requires: cryptography>=2.8]
    - aioasuswrt==1.3.1 [requires: asyncssh]
  - Authlib==0.15.2 [requires: cryptography]
    - pypoint==2.0.0 [requires: authlib]
  - broadlink==0.16.0 [requires: cryptography>=2.1.1]
  - HAP-python==3.0.0 [requires: cryptography]
  - hass-nabucasa==0.39.0 [requires: cryptography>=2.8,<4.0]
  - homeassistant==2020.12.0 [requires: cryptography==3.2]
  - http-ece==1.1.0 [requires: cryptography>=2.5]
    - pywebpush==1.9.2 [requires: http-ece>=1.1.0]
  - ihcsdk==2.7.0 [requires: cryptography]
  - josepy==1.5.0 [requires: cryptography>=0.8]
    - acme==1.9.0 [requires: josepy>=1.1.0]
      - hass-nabucasa==0.39.0 [requires: acme==1.9.0]
  - paramiko==2.7.2 [requires: cryptography>=2.5]
    - fortiosapi==0.10.8 [requires: paramiko]
  - py-vapid==1.7.1 [requires: cryptography>=2.5]
    - pywebpush==1.9.2 [requires: py-vapid>=1.4.0]
  - pyatv==0.7.5 [requires: cryptography>=2.6]
  - pyOpenSSL==20.0.0 [requires: cryptography>=3.2]
    - acme==1.9.0 [requires: PyOpenSSL>=0.15.1]
      - hass-nabucasa==0.39.0 [requires: acme==1.9.0]
    - josepy==1.5.0 [requires: PyOpenSSL>=0.13]
      - acme==1.9.0 [requires: josepy>=1.1.0]
        - hass-nabucasa==0.39.0 [requires: acme==1.9.0]
  - python-miio==0.5.4 [requires: cryptography>=3,<4]
  - python-telegram-bot==11.1.0 [requires: cryptography]
  - pywebpush==1.9.2 [requires: cryptography>=2.6.1]
  - PyXiaomiGateway==0.13.4 [requires: cryptography>=2.1.1]
  - SecretStorage==3.3.0 [requires: cryptography>=2.0]
    - keyring==21.2.0 [requires: SecretStorage>=3]
      - pyicloud==0.9.7 [requires: keyring>=8.0,<=9.3.1]
  - simplepush==1.1.4 [requires: cryptography]
  - snitun==0.20 [requires: cryptography>=2.5]
    - hass-nabucasa==0.39.0 [requires: snitun==0.20]

There are probably more - unfortunately pip-deptree is choking on quite a few packages so i suspect this list is incomplete.

It may allow other dependencies we can't currently consider to be considered - like e.g. hyperjson.

frenck commented 3 years ago

We pre-build wheels, so we can probably manage it for our docker based distros.

The concern is more the part of the community that relies on Core directly on a Python environment. They need to supply the toolchain (or, if they are lucky, the package has uploaded wheels matching their architecture).

Avoiding such dependencies has the preference in general, as it adds complexity for a specific group of users.

Jc2k commented 3 years ago

Indeed, it's unfortunate that such a core dependency is involved here...

Jc2k commented 3 years ago

I saw another cryptography version bump was merged. Luckily it was just a minor release. Not sure when 3.4 will be released. Linking to the current cryptography changelog.

frenck commented 3 years ago

Yes, but not sure if this is really a problem still?

Like cryptography delivers binary wheels for multiple architectures: https://pypi.org/project/cryptography/#files

(Which the orjson project didn't...)

For all Home Assistant installation based around containers, we build the wheels.

So, I'm not really sure what your case is right now?

Jc2k commented 3 years ago

Maybe i'm overthinking it 😁

I think we both agree that we will take cryptography 3.4 + its rust dependencies right? My thinking is that cryptography is a pretty core dependency of HA. It would be hard to migrate to something else. And it would be terrible to get stuck on an old version of a crypto library too. I don't really see a choice.

If thats the case we are de-facto saying "In the next few months it will be neccessary to have a working rust compiler to install home assistant from source", right? We've previously had architecture discussions about which install environments are supported, but fortunately the existing ADR seems to cover this new requirement with a "misc development tools" clause.

And wheels are great! And they dp help with this a lot. But I do have to disagree that wheels make the problem go away:

Someone still has to do the work to enable rust support on the HA wheel builder. I've skimmed the config and i can't see where it installs rust? Apologies if i overlooked something.
There aren't official wheels for ARM here?
There are wheels for Pi here but:
- they have been broken on stretch for last 2 releases
- they only target the python that Raspbian ships with and we currently target a minimum of 3.8. The wheels they have for 3.5 and 3.7 do not help us.
Given there aren't wheels for it, does this mean we need to write documentation for installing rust on here? Do we want to stop supporting Raspbian installs officially (and punt to community) because they are now much harder? Or maybe we have to raise the memory limits because rust needs lots of RAM? (I have no idea).

So to me, that feels like the kind of thing that people might like to have plenty of notice about? And it feels like there might be decisions to make because of it, and then work to do based on those. I'm aware that nothing needs to happen yet, but it seemed like something that was worth warning people about. Certainly i'd hope for plenty of notice if i was the maintainer for the wheel builder etc. Apologies if you'd rather wait for 3.4 to land or if there was a more appropriate venue?

frenck commented 3 years ago

Someone still has to do the work to enable rust support on the HA wheel builder.

Yep, it needs the rust build env. That should be addressable with a couple of extra packages in the build environment.

There aren't official wheels for ARM here?

That is not something we control. ARMv8 (aarch64) is provided by PyPI is provided at this point, however, since 3.4 isn't here yet... we simply don't know and have to wait until the moment is there.

Right now, there are too many things we don't know.

As for the PiWheels... yeah that project is problematic in general. Would be nice if they keep up with stuff. Would not recommend using that project in general, and honestly, out of our control.

Given there aren't wheels for it, does this mean we need to write documentation for installing rust on here?

We should remove more details from that page, not adding more. If one is not capable of setting up a Python environment like that, one should not be using that installation method.

Jc2k commented 3 years ago

I think the docs I linked to are already out of date and confusing as heck, so I definitely be on board with reducing them further or just deleting them.

I agree with your points about wheels, which is why i pointed out I don't think they are relevant to this discussions.

So AIUI:

You are fine with Home Assistant having a hard dep on Rust (yay)
We don't need to document this for virtualenv users
Theres no need to do any analysis about it up front
You want to wait until cryptography 3.4/3.5 is out to deal with it, there is no prep we can do up front.

I am fine with this, I just wanted to make sure it was understood. You know, @pvizeli might have had some concerns about Rust in the HA OS / wheels side of things. If you don't see a problem then lets close this.

frenck commented 3 years ago

You are fine with Home Assistant having a hard dep on Rust (yay)

Yay? 😄 I would not say so... but looking at upstream, it seems like we will not have a choice.

balloob commented 3 years ago

It's fine to have Rust available for our wheel builder 👍

Jc2k commented 3 years ago

FYI: This has now happened:

https://github.com/pyca/cryptography/blob/master/CHANGELOG.rst#34---2021-02-07

And is going about as well as expected with lots of entitled crap from the "community":

https://github.com/pyca/cryptography/issues/5771

Wheeeeee....

Antoni-Czaplicki commented 3 years ago

I am trying to add my own integration to the wheels-custom-integrations repository but I get a cryptography compilation error due to no compile option with rust. Is there any workaround for this?

Jc2k commented 3 years ago

Home Assistant itself currently only uses 3.3.1 (https://github.com/home-assistant/core/blob/dev/requirements.txt#L15) anyway.

When it upgrades to a newer cryptography I guess that build environment will either be updated to support rust or there will be a cryptography wheel available in it.

pvizeli commented 3 years ago

-> https://wheels.home-assistant.io/

home-assistant / architecture