aws / aws-encryption-sdk-cli

CLI wrapper around aws-encryption-sdk-python
Apache License 2.0
80 stars 39 forks source link

aws-encryption-sdk-python v3.1.1 breaks aws-encryption-cli #307

Open mpalmer opened 2 years ago

mpalmer commented 2 years ago

Problem:

Simple command, taken straight from the fine manual:

keyArn=<etc etc>
aws-encryption-cli --encrypt \
                     --input hello.txt \
                     --wrapping-keys key=$keyArn \
                     --metadata-output /tmp/metadata \
                     --encryption-context purpose=test \
                     --commitment-policy require-encrypt-require-decrypt \
                     --output .

When run with aws-encryption-sdk v3.1.0, this command succeeds. When run with aws-encryption-sdk v3.1.1 (ie after running pip install aws-encryption-sdk==3.1.1), I get this output and a failure exit status:

2022-08-01 07:30:03,179 - MainThread - aws_encryption_sdk_cli - WARNING - Operation failed: deleting output file: ./hello.txt.encrypted
Encountered unexpected error: increase verbosity to see details.
NotSupportedError("Unsupported signing algorithm info")

Chucking in ALL THE VERBOSE (-vvvv) I get this:

2022-08-01 07:30:54,911 - MainThread - aws_encryption_sdk_cli - DEBUG - Encryption mode: encrypt
2022-08-01 07:30:54,912 - MainThread - aws_encryption_sdk_cli - DEBUG - Encryption source: hello.txt
2022-08-01 07:30:54,912 - MainThread - aws_encryption_sdk_cli - DEBUG - Encryption destination: .
2022-08-01 07:30:54,912 - MainThread - aws_encryption_sdk_cli - DEBUG - Wrapping key provider configuration: [{'key': ['<etc etc>'], 'provider': 'aws-encryption-sdk-cli::aws-kms', 'discovery': False}]
2022-08-01 07:30:54,912 - MainThread - aws_encryption_sdk_cli - DEBUG - Suffix requested: None
2022-08-01 07:30:54,912 - MainThread - aws_encryption_sdk_cli - DEBUG - Loading provider: aws-encryption-sdk-cli::aws-kms
2022-08-01 07:30:54,912 - MainThread - aws_encryption_sdk_cli - DEBUG - Discovering master key provider plugins
2022-08-01 07:30:54,913 - MainThread - aws_encryption_sdk_cli - INFO - Collecting plugin "aws-kms" registered by "aws-encryption-sdk-cli 4.1.0"
2022-08-01 07:30:54,913 - MainThread - aws_encryption_sdk_cli - DEBUG - Plugin details: {'name': 'aws-kms', 'module_name': 'aws_encryption_sdk_cli.key_providers', 'attrs': ('aws_kms_master_key_provider',), 'extras': (), 'dist': aws-encryption-sdk-cli 4.1.0 (/usr/local/lib/python3.9/dist-packages)}
2022-08-01 07:30:54,952 - MainThread - botocore.hooks - DEBUG - Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2022-08-01 07:30:54,954 - MainThread - botocore.hooks - DEBUG - Changing event name from before-call.apigateway to before-call.api-gateway
2022-08-01 07:30:54,955 - MainThread - botocore.hooks - DEBUG - Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2022-08-01 07:30:54,956 - MainThread - botocore.hooks - DEBUG - Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2022-08-01 07:30:54,956 - MainThread - botocore.hooks - DEBUG - Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2022-08-01 07:30:54,957 - MainThread - botocore.hooks - DEBUG - Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2022-08-01 07:30:54,958 - MainThread - botocore.hooks - DEBUG - Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2022-08-01 07:30:54,960 - MainThread - botocore.hooks - DEBUG - Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2022-08-01 07:30:54,960 - MainThread - botocore.hooks - DEBUG - Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2022-08-01 07:30:54,961 - MainThread - botocore.hooks - DEBUG - Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2022-08-01 07:30:54,961 - MainThread - botocore.hooks - DEBUG - Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2022-08-01 07:30:54,964 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2022-08-01 07:30:54,965 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2022-08-01 07:30:54,966 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2022-08-01 07:30:54,967 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2022-08-01 07:30:54,967 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2022-08-01 07:30:54,967 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2022-08-01 07:30:54,969 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
2022-08-01 07:30:54,970 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2022-08-01 07:30:54,971 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 27
2022-08-01 07:30:54,971 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2022-08-01 07:30:54,972 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/<redacted> HTTP/1.1" 200 1458
2022-08-01 07:30:54,974 - MainThread - botocore.credentials - DEBUG - Found credentials from IAM Role: <redacted>
2022-08-01 07:30:54,975 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/endpoints.json
2022-08-01 07:30:55,000 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/sdk-default-configuration.json
2022-08-01 07:30:55,001 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7ffa9c5f3c10>
2022-08-01 07:30:55,008 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/kms/2014-11-01/service-2.json
2022-08-01 07:30:55,012 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.kms: calling handler <function add_generate_presigned_url at 0x7ffa9c5fe4c0>
2022-08-01 07:30:55,014 - MainThread - botocore.endpoint - DEBUG - Setting kms timeout as (60, 60)
2022-08-01 07:30:55,015 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.9/dist-packages/botocore/data/_retry.json
2022-08-01 07:30:55,015 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: kms
2022-08-01 07:30:55,016 - MainThread - aws_encryption_sdk_cli - DEBUG - Requested source: hello.txt
2022-08-01 07:30:55,016 - MainThread - aws_encryption_sdk_cli - DEBUG - Expanded source: ['hello.txt']
2022-08-01 07:30:55,016 - MainThread - aws_encryption_sdk_cli - DEBUG - Duplicating filename hello.txt into .
2022-08-01 07:30:55,016 - MainThread - aws_encryption_sdk_cli - INFO - encrypting file hello.txt to ./hello.txt.encrypted
2022-08-01 07:30:55,017 - MainThread - aws_encryption_sdk.materials_managers.default - DEBUG - Generating signing key
2022-08-01 07:30:55,017 - MainThread - aws_encryption_sdk.streaming_client - DEBUG - Closing stream
2022-08-01 07:30:55,017 - MainThread - aws_encryption_sdk_cli - WARNING - Operation failed: deleting output file: ./hello.txt.encrypted
2022-08-01 07:30:55,017 - MainThread - aws_encryption_sdk_cli - DEBUG - Encountered unexpected error: increase verbosity to see details.
NotSupportedError("Unsupported signing algorithm info")
2022-08-01 07:30:55,018 - MainThread - aws_encryption_sdk_cli - DEBUG - Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk_cli/__init__.py", line 288, in cli
    process_cli_request(stream_args, args)         
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk_cli/__init__.py", line 225, in process_cli_request
    handler.process_single_file(stream_args=stream_args, source=_source, destination=_destination)
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk_cli/internal/io_handling.py", line 370, in process_single_file
    operation_result = self.process_single_operation(
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk_cli/internal/io_handling.py", line 302, in process_single_operation
    return self._single_io_write(                  
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk_cli/internal/io_handling.py", line 240, in _single_io_write
    header=json_ready_header(handler.header),      
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk/streaming_client.py", line 218, in header
    self._prep_message()                           
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk/streaming_client.py", line 462, in _prep_message
    self._encryption_materials = self.config.materials_manager.get_encryption_materials(
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk/materials_managers/default.py", line 87, in get_encryption_materials
    signing_key = self._generate_signing_key_and_update_encryption_context(algorithm, encryption_context)
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk/materials_managers/default.py", line 58, in _generate_signing_key_and_update_encryption_context
    signer = Signer(algorithm=algorithm, key=generate_ecc_signing_key(algorithm=algorithm))
  File "/usr/local/lib/python3.9/dist-packages/aws_encryption_sdk/internal/crypto/elliptic_curve.py", line 185, in generate_ecc_signing_key
    raise NotSupportedError("Unsupported signing algorithm info")
aws_encryption_sdk.exceptions.NotSupportedError: Unsupported signing algorithm info

Encountered unexpected error: increase verbosity to see details.
NotSupportedError("Unsupported signing algorithm info")

Running an up-to-date Debian 11 ("Bullseye") system in an EC2 instance, built from the official Debian AMI. Nothing particularly fancy in the machine that I can think of -- no fancy env vars, customised config, etc.

texastony commented 2 years ago

Howdy @mpalmer, Thank you for bringing this to our attention.

I attempted to recreate your issue on a Mac Laptop.

    mkdir temp; cd temp;
    python -m venv venv; source venv/bin/activate;
    python -m pip install aws-encryption-sdk-cli==4.1.0;
    echo “hello world” > hello.txt;
    // action to fetch AWS credentials
    keyArn=$AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID;
    aws-encryption-cli --encrypt \
                       --input hello.txt \
                       --wrapping-keys key=$keyArn \
                       --metadata-output /tmp/metadata \
                       --encryption-context purpose=test \
                       --commitment-policy require-encrypt-require-decrypt \
                       --output .
    // encrypt succeeded
    aws-encryption-cli --decrypt \
                       --input hello.txt.encrypted \
                       --wrapping-keys key=$keyArn \
                       --metadata-output /tmp/metadata \
                       --encryption-context purpose=test \
                       --commitment-policy require-encrypt-require-decrypt \
                       --output .
    // decrypt succeeded
    // Check aws-encryption-sdk version
    python -m pip list | grep “aws-encryption”
    // output: aws-encryption-sdk     3.1.1
    // output: aws-encryption-sdk-cli 4.1.0

Trying this on a Debian EC2 Instance:

    sudo apt update; sudo apt full-upgrade;
    sudo apt-get install python3-venv;
    mkdir temp; cd temp;
    python3 -m venv venv; // The rest of the script from above.

Both trials succeeded. In both cases, python -m pip list | grep “aws-encryption” reported the same versions of the ESDK-Python and ESDK-CLI that you are using.

On Debian, I used Python 3.9.2. On Mac, I used Python 3.9.13.

So I cannot immediately re-create your issue.

Can you tell me what version of Cryptography you are using? python -m pip show cryptography

Can you tell me what version of Python3.9 you are using?

From your stack-trace, I see that you are using the system wide Python. Can you try running the ESDK-CLI in a clean Python virtual environment?

It could be that other packages on your host are requiring a different version of Cryptography or other dependency that is interfering with the ESDK’s dependencies.

This issue is pending a reply from @mpalmer answering the above questions.

Much Obliged, AWS Crypto Tools

mpalmer commented 2 years ago

How odd. Here's the full dpkg and pip package lists:

# dpkg -l |grep python
ii  libpython3-dev:amd64                3.9.2-3                        amd64        header files and a static library for Python (default)
ii  libpython3-stdlib:amd64             3.9.2-3                        amd64        interactive high-level object-oriented language (default python3 version)
ii  libpython3.9:amd64                  3.9.2-1                        amd64        Shared Python runtime library (version 3.9)
ii  libpython3.9-dev:amd64              3.9.2-1                        amd64        Header files and a static library for Python (v3.9)
ii  libpython3.9-minimal:amd64          3.9.2-1                        amd64        Minimal subset of the Python language (version 3.9)
ii  libpython3.9-stdlib:amd64           3.9.2-1                        amd64        Interactive high-level object-oriented language (standard library, version 3.9)
ii  python-apt-common                   2.2.1                          all          Python interface to libapt-pkg (locales)
ii  python-pip-whl                      20.3.4-4+deb11u1               all          Python package installer (pip wheels)
ii  python3                             3.9.2-3                        amd64        interactive high-level object-oriented language (default python3 version)
ii  python3-apt                         2.2.1                          amd64        Python 3 interface to libapt-pkg
ii  python3-attr                        20.3.0-1                       all          Attributes without boilerplate (Python 3)
ii  python3-blinker                     1.4+dfsg1-0.3                  all          fast, simple object-to-object and broadcast signaling library
ii  python3-boto                        2.49.0-3                       all          Python interface to Amazon's Web Services - Python 3.x
ii  python3-botocore                    1.20.0+repack-1                all          Low-level, data-driven core of boto 3 (Python 3)
ii  python3-certifi                     2020.6.20-1                    all          root certificates for validating SSL certs and verifying TLS hosts (python3)
ii  python3-cffi-backend:amd64          1.14.5-1                       amd64        Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-chardet                     4.0.0-1                        all          universal character encoding detector for Python3
ii  python3-colorama                    0.4.4-1                        all          Cross-platform colored terminal text in Python - Python 3.x
ii  python3-configobj                   5.0.6-4                        all          simple but powerful config file reader and writer for Python 3
ii  python3-cryptography                3.3.2-1                        amd64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-dateutil                    2.8.1-6                        all          powerful extensions to the standard Python 3 datetime module
ii  python3-dbus                        1.2.16-5                       amd64        simple interprocess messaging system (Python 3 interface)
ii  python3-debconf                     1.5.77                         all          interact with debconf from Python 3
ii  python3-debian                      0.1.39                         all          Python 3 modules to work with Debian-related data formats
ii  python3-debianbts                   3.1.0                          all          Python interface to Debian's Bug Tracking System
ii  python3-dev                         3.9.2-3                        amd64        header files and a static library for Python (default)
ii  python3-distro-info                 1.0                            all          information about distributions' releases (Python 3 module)
ii  python3-distutils                   3.9.2-1                        all          distutils package for Python 3.x
ii  python3-docutils                    0.16+dfsg-4                    all          text processing system for reStructuredText (implemented in Python 3)
ii  python3-httplib2                    0.18.1-3                       all          comprehensive HTTP client library written for Python3
ii  python3-idna                        2.10-1                         all          Python IDNA2008 (RFC 5891) handling (Python 3)
ii  python3-importlib-metadata          1.6.0-2                        all          library to access the metadata for a Python package - Python 3.x
ii  python3-jinja2                      2.11.3-1                       all          small but fast and easy to use stand-alone template engine
ii  python3-jmespath                    0.10.0-1                       all          JSON Matching Expressions (Python 3)
ii  python3-json-pointer                2.0-2                          all          resolve JSON pointers - Python 3.x
ii  python3-jsonpatch                   1.25-3                         all          library to apply JSON patches - Python 3.x
ii  python3-jsonschema                  3.2.0-3                        all          An(other) implementation of JSON Schema (Draft 3 and 4) - Python 3.x
ii  python3-jwt                         1.7.1-2                        all          Python 3 implementation of JSON Web Token
ii  python3-lib2to3                     3.9.2-1                        all          Interactive high-level object-oriented language (lib2to3)
ii  python3-markupsafe                  1.1.1-1+b3                     amd64        HTML/XHTML/XML string library for Python 3
ii  python3-minimal                     3.9.2-3                        amd64        minimal subset of the Python language (default python3 version)
ii  python3-more-itertools              4.2.0-3                        all          library with routines for operating on iterables, beyond itertools (Python 3)
ii  python3-oauthlib                    3.1.0-2                        all          generic, spec-compliant implementation of OAuth for Python3
ii  python3-pip                         20.3.4-4+deb11u1               all          Python package installer
ii  python3-pkg-resources               52.0.0-4                       all          Package Discovery and Resource Access using pkg_resources
ii  python3-pyasn1                      0.4.8-1                        all          ASN.1 library for Python (Python 3 module)
ii  python3-pycurl                      7.43.0.6-5                     amd64        Python bindings to libcurl (Python 3)
ii  python3-pyrsistent:amd64            0.15.5-1+b3                    amd64        persistent/functional/immutable data structures for Python
ii  python3-pysimplesoap                1.16.2-3                       all          simple and lightweight SOAP Library (Python 3)
ii  python3-reportbug                   7.10.3+deb11u1                 all          Python modules for interacting with bug tracking systems
ii  python3-requests                    2.25.1+dfsg-2                  all          elegant and simple HTTP library for Python3, built for human beings
ii  python3-roman                       2.0.0-5                        all          module for generating/analyzing Roman numerals for Python 3
ii  python3-rsa                         4.0-4                          all          Pure-Python RSA implementation (Python 3)
ii  python3-s3transfer                  0.3.4-1                        all          Amazon S3 Transfer Manager for Python3
ii  python3-setuptools                  52.0.0-4                       all          Python3 Distutils Enhancements
ii  python3-six                         1.16.0-2                       all          Python 2 and 3 compatibility library (Python 3 interface)
ii  python3-urllib3                     1.26.5-1~exp1                  all          HTTP library with thread-safe connection pooling for Python3
ii  python3-wheel                       0.34.2-1                       all          built-package format for Python
ii  python3-yaml                        5.3.1-5                        amd64        YAML parser and emitter for Python3
ii  python3-zipp                        1.0.0-3                        all          pathlib-compatible Zipfile object wrapper - Python 3.x
ii  python3.9                           3.9.2-1                        amd64        Interactive high-level object-oriented language (version 3.9)
ii  python3.9-dev                       3.9.2-1                        amd64        Header files and a static library for Python (v3.9)
ii  python3.9-minimal                   3.9.2-1                        amd64        Minimal subset of the Python language (version 3.9)
# pip list
Package                Version
---------------------- --------------
attrs                  20.3.0
aws-encryption-sdk     3.1.1
aws-encryption-sdk-cli 4.1.0
awscli                 1.19.1
base64io               1.0.3
blinker                1.4
boto                   2.49.0
boto3                  1.24.46
botocore               1.27.46
certifi                2020.6.20
chardet                4.0.0
cloud-init             20.4.1
colorama               0.4.4
configobj              5.0.6
cryptography           3.3.2
dbus-python            1.2.16
distro-info            1.0
docutils               0.16
httplib2               0.18.1
idna                   2.10
importlib-metadata     1.6.0
Jinja2                 2.11.3
jmespath               0.10.0
jsonpatch              1.25
jsonpointer            2.0
jsonschema             3.2.0
MarkupSafe             1.1.1
more-itertools         4.2.0
oauthlib               3.1.0
pip                    20.3.4
pyasn1                 0.4.8
pycurl                 7.43.0.6
PyJWT                  1.7.1
pyrsistent             0.15.5
PySimpleSOAP           1.16.2
python-apt             2.2.1
python-dateutil        2.8.1
python-debian          0.1.39
python-debianbts       3.1.0
PyYAML                 5.3.1
reportbug              7.10.3+deb11u1
requests               2.25.1
roman                  2.0.0
rsa                    4.0
s3transfer             0.6.0
setuptools             52.0.0
six                    1.16.0
unattended-upgrades    0.1
urllib3                1.26.5
wheel                  0.34.2
wrapt                  1.14.1
zipp                   1.0.0
# pip show cryptography
Name: cryptography
Version: 3.3.2
Summary: cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Home-page: https://github.com/pyca/cryptography
Author: The cryptography developers
Author-email: cryptography-dev@python.org
License: BSD or Apache License, Version 2.0
Location: /usr/lib/python3/dist-packages
Requires: 
Required-by: aws-encryption-sdk

Running in a venv does seem to work, and the package list is... interestingly different:

(venv) # pip list
Package                Version
---------------------- -------
attrs                  22.1.0
aws-encryption-sdk     3.1.1
aws-encryption-sdk-cli 4.1.0
base64io               1.0.3
boto3                  1.24.48
botocore               1.27.48
cffi                   1.15.1
cryptography           37.0.4
jmespath               1.0.1
pip                    20.3.4
pkg-resources          0.0.0
pycparser              2.21
python-dateutil        2.8.2
s3transfer             0.6.0
setuptools             44.1.1
six                    1.16.0
urllib3                1.26.11
wrapt                  1.14.1

That is a very different version of the cryptography package. If I downgrade the version of cryptography in the venv, the problem comes back:

(venv) # pip install cryptography==3.3.2
[...]
(venv) # aws-encryption-cli --encrypt
[...]
Encountered unexpected error: increase verbosity to see details.
NotSupportedError("Unsupported signing algorithm info")

So, the problem appears to be that a change that went into aws-encryption-sdk v3.1.1 now requires not-entirely-ancient versions of cryptography. Doesn't look like that's going to make it into Debian any time in the next week or so. Maybe chuck a stricter version constraint for cryptography into aws-encryption-sdk. That's probably more worthwhile than trying to maintain compatibility with cryptography versions from the dark ages.