search

confluentinc / confluent-docker-utils

Common Python utils for testing Confluent's Docker images
Apache License 2.0
3 stars 5 forks source link

relax requirements to allow debian packages on arm64 #27

Closed xvrl closed 3 years ago

xvrl commented 3 years ago

Some of our python dependencies depend on native libraries, which do not exist in pre-compiled form for arm64 in PyPI.

To avoid having to recompile python packages from scratch on arm64, we relax our requirements to allow using the python libraries already included with the base image we use in cloud.

xvrl commented 3 years ago

There are 2 issues to address here. First, moving to earlier versions -- did you do an audit to make sure there aren't, e.g., cves?

I checked Jinja2, boto, docker, docker-compose did not have any CVEs for the versions listed above. cryptography 2.6.1 doesn't have any additional CVEs compared to 2.7.

tbh, it's a bit surprising that we need to go to older versions to get arm64 support, I would have expected projects add support and it just happens via CI after it's added. Do we understand why so many dependencies seem to have spotty support for arm64 after adding support in a specific release?

Debian has had arm64 support for a long time and builds their own packages. The same may not be true for the upstream package maintainers uploading to PyPI.

Second, I think we've never actually come to a conclusion on using >= in python deps. The main objection is that it makes builds unreproducible, but with the obvious benefit that you get updates for free. To date, I think the former issue has won out over the latter, but I think its open to discussion.

While using == in our builds might make it somewhat more reproducible, I doubt all the dependencies we pull in transitively have been pinned in the packages we depend on. It might be more predictable to pin the packages directly when installing the package using pip3 install -r <override-reqs>?

It might be possible to further reduce the set of packages we relax by looking at which ones do ship binaries or not. I can look into that.

xvrl commented 3 years ago

@ewencp I was able to reduce the set of affected libraries and limit the "downgrade" to cryptography which along with paramiko has binary dependencies. Only cryptography 2.6.1 with additional security patches is available in debian stable.

niteshmor commented 3 years ago

Slightly late to the thread, but there have been some field requests to upgrade python cryptography to 3.2 in order to resolve this https://github.com/advisories/GHSA-hggm-jpg3-v476 cc @xvrl

xvrl commented 3 years ago

@niteshmor >=2.6.1 should not prevent you from pulling in 3.2. There could however be other transitive dependencies that will not work with 3.2 since it includes many backwards incompatible changes.

niteshmor commented 3 years ago

@niteshmor >=2.6.1 should not prevent you from pulling in 3.2. There could however be other transitive dependencies that will not work with 3.2 since it includes many backwards incompatible changes.

Ack! This was mostly just a "hey, we have people asking things to be upgraded, so this may need to be revisited soon" thing. Agree with more due diligence necessary to consider with the upgrade to 3.2.