search

aws-cloudformation / cloudformation-cli-python-plugin

The CloudFormation Provider Development Toolkit Python Plugin allows you to autogenerate Python code based on an input schema.
Apache License 2.0
108 stars 49 forks source link

OOTB generated python CF resource cannot be used with cfn test #247

Open carpnick opened 1 year ago

carpnick commented 1 year ago

Repro:

Environment:

Research:

carpnick commented 1 year ago

Full trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/task/dd_dd_test1/handlers.py", line 3, in <module>
    from cloudformation_cli_python_lib import (
  File "/var/task/cloudformation_cli_python_lib/__init__.py", line 4, in <module>
    from .hook import Hook  # noqa: F401
  File "/var/task/cloudformation_cli_python_lib/hook.py", line 9, in <module>
    from .cipher import Cipher, KmsCipher
  File "/var/task/cloudformation_cli_python_lib/cipher.py", line 4, in <module>
    import aws_encryption_sdk  # type: ignore
  File "/var/task/aws_encryption_sdk/__init__.py", line 19, in <module>
    from aws_encryption_sdk.caches.local import LocalCryptoMaterialsCache  # noqa
  File "/var/task/aws_encryption_sdk/caches/__init__.py", line 25, in <module>
    from ..internal.formatting.encryption_context import serialize_encryption_context
  File "/var/task/aws_encryption_sdk/internal/formatting/__init__.py", line 14, in <module>
    from .serialize import serialize_header
  File "/var/task/aws_encryption_sdk/internal/formatting/serialize.py", line 17, in <module>
    import aws_encryption_sdk.internal.defaults
  File "/var/task/aws_encryption_sdk/internal/defaults.py", line 16, in <module>
    import aws_encryption_sdk.identifiers
  File "/var/task/aws_encryption_sdk/identifiers.py", line 18, in <module>
    from cryptography.hazmat.primitives.asymmetric import ec, padding, rsa
  File "/var/task/cryptography/hazmat/primitives/asymmetric/ec.py", line 10, in <module>
    from cryptography.hazmat._oid import ObjectIdentifier
  File "/var/task/cryptography/hazmat/_oid.py", line 7, in <module>
    from cryptography.hazmat.bindings._rust import (
ImportError: cannot import name 'ObjectIdentifier' from 'cryptography.hazmat.bindings._rust' (unknown location)
carpnick commented 1 year ago

Found out issue:

1st issue: Template assumes X86. If you are on an Apple Mac M1, you have to modify the SAM template to use arm64 using the Architectures property. User error.

Enhancement Request - With cfn init ask user architecture and support arm64 and x86/AMD64 and modify templating of new resources as required.

2nd issue: Fundamentally we have a broken workflow I think. Any python package that requires native install cannot support the copysource workflow that SAM supports since it doesnt copy over all files. It excludes certain files we need for native packages. Broken because of this line more than likely.

Confirmed:

2023-01-17 20:04:34,335 | File (_padding.abi3.so) is in ignored set, skipping it
2023-01-17 20:04:34,335 | File (_openssl.abi3.so) is in ignored set, skipping it

This fundamentally breaks native zipping with SAM CLI to allow contract testing locally. Posting workaround below.

AWS team - is there any plans to make it so we can ship native docker container images as an integration instead? If not, how will we overcome the native dependency install? SAM CLI Change? aws-lambda-builders change? In this case we cannot say not supported since cryptography is in the cloudformation-cli-python-plugin dependency tree, let alone customers having their own dependencies.

As I see it - this completely blocks cfn test use cases running locally with sam local start-lambda using the documented way of running sam build and the templated default that comes from cfn init.

Complete workaround: