Open mpalmer opened 2 years ago
Howdy @mpalmer, Thank you for bringing this to our attention.
I attempted to recreate your issue on a Mac Laptop.
mkdir temp; cd temp;
python -m venv venv; source venv/bin/activate;
python -m pip install aws-encryption-sdk-cli==4.1.0;
echo “hello world” > hello.txt;
// action to fetch AWS credentials
keyArn=$AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID;
aws-encryption-cli --encrypt \
--input hello.txt \
--wrapping-keys key=$keyArn \
--metadata-output /tmp/metadata \
--encryption-context purpose=test \
--commitment-policy require-encrypt-require-decrypt \
--output .
// encrypt succeeded
aws-encryption-cli --decrypt \
--input hello.txt.encrypted \
--wrapping-keys key=$keyArn \
--metadata-output /tmp/metadata \
--encryption-context purpose=test \
--commitment-policy require-encrypt-require-decrypt \
--output .
// decrypt succeeded
// Check aws-encryption-sdk version
python -m pip list | grep “aws-encryption”
// output: aws-encryption-sdk 3.1.1
// output: aws-encryption-sdk-cli 4.1.0
Trying this on a Debian EC2 Instance:
sudo apt update; sudo apt full-upgrade;
sudo apt-get install python3-venv;
mkdir temp; cd temp;
python3 -m venv venv; // The rest of the script from above.
Both trials succeeded. In both cases, python -m pip list | grep “aws-encryption”
reported the
same versions of the ESDK-Python and ESDK-CLI that you are using.
On Debian, I used Python 3.9.2. On Mac, I used Python 3.9.13.
So I cannot immediately re-create your issue.
Can you tell me what version of Cryptography you are using?
python -m pip show cryptography
Can you tell me what version of Python3.9 you are using?
From your stack-trace, I see that you are using the system wide Python. Can you try running the ESDK-CLI in a clean Python virtual environment?
It could be that other packages on your host are requiring a different version of Cryptography or other dependency that is interfering with the ESDK’s dependencies.
This issue is pending a reply from @mpalmer answering the above questions.
Much Obliged, AWS Crypto Tools
How odd. Here's the full dpkg and pip package lists:
# dpkg -l |grep python
ii libpython3-dev:amd64 3.9.2-3 amd64 header files and a static library for Python (default)
ii libpython3-stdlib:amd64 3.9.2-3 amd64 interactive high-level object-oriented language (default python3 version)
ii libpython3.9:amd64 3.9.2-1 amd64 Shared Python runtime library (version 3.9)
ii libpython3.9-dev:amd64 3.9.2-1 amd64 Header files and a static library for Python (v3.9)
ii libpython3.9-minimal:amd64 3.9.2-1 amd64 Minimal subset of the Python language (version 3.9)
ii libpython3.9-stdlib:amd64 3.9.2-1 amd64 Interactive high-level object-oriented language (standard library, version 3.9)
ii python-apt-common 2.2.1 all Python interface to libapt-pkg (locales)
ii python-pip-whl 20.3.4-4+deb11u1 all Python package installer (pip wheels)
ii python3 3.9.2-3 amd64 interactive high-level object-oriented language (default python3 version)
ii python3-apt 2.2.1 amd64 Python 3 interface to libapt-pkg
ii python3-attr 20.3.0-1 all Attributes without boilerplate (Python 3)
ii python3-blinker 1.4+dfsg1-0.3 all fast, simple object-to-object and broadcast signaling library
ii python3-boto 2.49.0-3 all Python interface to Amazon's Web Services - Python 3.x
ii python3-botocore 1.20.0+repack-1 all Low-level, data-driven core of boto 3 (Python 3)
ii python3-certifi 2020.6.20-1 all root certificates for validating SSL certs and verifying TLS hosts (python3)
ii python3-cffi-backend:amd64 1.14.5-1 amd64 Foreign Function Interface for Python 3 calling C code - runtime
ii python3-chardet 4.0.0-1 all universal character encoding detector for Python3
ii python3-colorama 0.4.4-1 all Cross-platform colored terminal text in Python - Python 3.x
ii python3-configobj 5.0.6-4 all simple but powerful config file reader and writer for Python 3
ii python3-cryptography 3.3.2-1 amd64 Python library exposing cryptographic recipes and primitives (Python 3)
ii python3-dateutil 2.8.1-6 all powerful extensions to the standard Python 3 datetime module
ii python3-dbus 1.2.16-5 amd64 simple interprocess messaging system (Python 3 interface)
ii python3-debconf 1.5.77 all interact with debconf from Python 3
ii python3-debian 0.1.39 all Python 3 modules to work with Debian-related data formats
ii python3-debianbts 3.1.0 all Python interface to Debian's Bug Tracking System
ii python3-dev 3.9.2-3 amd64 header files and a static library for Python (default)
ii python3-distro-info 1.0 all information about distributions' releases (Python 3 module)
ii python3-distutils 3.9.2-1 all distutils package for Python 3.x
ii python3-docutils 0.16+dfsg-4 all text processing system for reStructuredText (implemented in Python 3)
ii python3-httplib2 0.18.1-3 all comprehensive HTTP client library written for Python3
ii python3-idna 2.10-1 all Python IDNA2008 (RFC 5891) handling (Python 3)
ii python3-importlib-metadata 1.6.0-2 all library to access the metadata for a Python package - Python 3.x
ii python3-jinja2 2.11.3-1 all small but fast and easy to use stand-alone template engine
ii python3-jmespath 0.10.0-1 all JSON Matching Expressions (Python 3)
ii python3-json-pointer 2.0-2 all resolve JSON pointers - Python 3.x
ii python3-jsonpatch 1.25-3 all library to apply JSON patches - Python 3.x
ii python3-jsonschema 3.2.0-3 all An(other) implementation of JSON Schema (Draft 3 and 4) - Python 3.x
ii python3-jwt 1.7.1-2 all Python 3 implementation of JSON Web Token
ii python3-lib2to3 3.9.2-1 all Interactive high-level object-oriented language (lib2to3)
ii python3-markupsafe 1.1.1-1+b3 amd64 HTML/XHTML/XML string library for Python 3
ii python3-minimal 3.9.2-3 amd64 minimal subset of the Python language (default python3 version)
ii python3-more-itertools 4.2.0-3 all library with routines for operating on iterables, beyond itertools (Python 3)
ii python3-oauthlib 3.1.0-2 all generic, spec-compliant implementation of OAuth for Python3
ii python3-pip 20.3.4-4+deb11u1 all Python package installer
ii python3-pkg-resources 52.0.0-4 all Package Discovery and Resource Access using pkg_resources
ii python3-pyasn1 0.4.8-1 all ASN.1 library for Python (Python 3 module)
ii python3-pycurl 7.43.0.6-5 amd64 Python bindings to libcurl (Python 3)
ii python3-pyrsistent:amd64 0.15.5-1+b3 amd64 persistent/functional/immutable data structures for Python
ii python3-pysimplesoap 1.16.2-3 all simple and lightweight SOAP Library (Python 3)
ii python3-reportbug 7.10.3+deb11u1 all Python modules for interacting with bug tracking systems
ii python3-requests 2.25.1+dfsg-2 all elegant and simple HTTP library for Python3, built for human beings
ii python3-roman 2.0.0-5 all module for generating/analyzing Roman numerals for Python 3
ii python3-rsa 4.0-4 all Pure-Python RSA implementation (Python 3)
ii python3-s3transfer 0.3.4-1 all Amazon S3 Transfer Manager for Python3
ii python3-setuptools 52.0.0-4 all Python3 Distutils Enhancements
ii python3-six 1.16.0-2 all Python 2 and 3 compatibility library (Python 3 interface)
ii python3-urllib3 1.26.5-1~exp1 all HTTP library with thread-safe connection pooling for Python3
ii python3-wheel 0.34.2-1 all built-package format for Python
ii python3-yaml 5.3.1-5 amd64 YAML parser and emitter for Python3
ii python3-zipp 1.0.0-3 all pathlib-compatible Zipfile object wrapper - Python 3.x
ii python3.9 3.9.2-1 amd64 Interactive high-level object-oriented language (version 3.9)
ii python3.9-dev 3.9.2-1 amd64 Header files and a static library for Python (v3.9)
ii python3.9-minimal 3.9.2-1 amd64 Minimal subset of the Python language (version 3.9)
# pip list
Package Version
---------------------- --------------
attrs 20.3.0
aws-encryption-sdk 3.1.1
aws-encryption-sdk-cli 4.1.0
awscli 1.19.1
base64io 1.0.3
blinker 1.4
boto 2.49.0
boto3 1.24.46
botocore 1.27.46
certifi 2020.6.20
chardet 4.0.0
cloud-init 20.4.1
colorama 0.4.4
configobj 5.0.6
cryptography 3.3.2
dbus-python 1.2.16
distro-info 1.0
docutils 0.16
httplib2 0.18.1
idna 2.10
importlib-metadata 1.6.0
Jinja2 2.11.3
jmespath 0.10.0
jsonpatch 1.25
jsonpointer 2.0
jsonschema 3.2.0
MarkupSafe 1.1.1
more-itertools 4.2.0
oauthlib 3.1.0
pip 20.3.4
pyasn1 0.4.8
pycurl 7.43.0.6
PyJWT 1.7.1
pyrsistent 0.15.5
PySimpleSOAP 1.16.2
python-apt 2.2.1
python-dateutil 2.8.1
python-debian 0.1.39
python-debianbts 3.1.0
PyYAML 5.3.1
reportbug 7.10.3+deb11u1
requests 2.25.1
roman 2.0.0
rsa 4.0
s3transfer 0.6.0
setuptools 52.0.0
six 1.16.0
unattended-upgrades 0.1
urllib3 1.26.5
wheel 0.34.2
wrapt 1.14.1
zipp 1.0.0
# pip show cryptography
Name: cryptography
Version: 3.3.2
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The cryptography developers
Author-email: cryptography-dev@python.org
License: BSD or Apache License, Version 2.0
Location: /usr/lib/python3/dist-packages
Requires:
Required-by: aws-encryption-sdk
Running in a venv does seem to work, and the package list is... interestingly different:
(venv) # pip list
Package Version
---------------------- -------
attrs 22.1.0
aws-encryption-sdk 3.1.1
aws-encryption-sdk-cli 4.1.0
base64io 1.0.3
boto3 1.24.48
botocore 1.27.48
cffi 1.15.1
cryptography 37.0.4
jmespath 1.0.1
pip 20.3.4
pkg-resources 0.0.0
pycparser 2.21
python-dateutil 2.8.2
s3transfer 0.6.0
setuptools 44.1.1
six 1.16.0
urllib3 1.26.11
wrapt 1.14.1
That is a very different version of the cryptography
package. If I downgrade the version of cryptography
in the venv, the problem comes back:
(venv) # pip install cryptography==3.3.2
[...]
(venv) # aws-encryption-cli --encrypt
[...]
Encountered unexpected error: increase verbosity to see details.
NotSupportedError("Unsupported signing algorithm info")
So, the problem appears to be that a change that went into aws-encryption-sdk
v3.1.1 now requires not-entirely-ancient versions of cryptography
. Doesn't look like that's going to make it into Debian any time in the next week or so. Maybe chuck a stricter version constraint for cryptography
into aws-encryption-sdk
. That's probably more worthwhile than trying to maintain compatibility with cryptography
versions from the dark ages.
Problem:
Simple command, taken straight from the fine manual:
When run with
aws-encryption-sdk
v3.1.0, this command succeeds. When run withaws-encryption-sdk
v3.1.1 (ie after runningpip install aws-encryption-sdk==3.1.1
), I get this output and a failure exit status:Chucking in ALL THE VERBOSE (
-vvvv
) I get this:Running an up-to-date Debian 11 ("Bullseye") system in an EC2 instance, built from the official Debian AMI. Nothing particularly fancy in the machine that I can think of -- no fancy env vars, customised config, etc.