Bug: python 3.12 with setuptools installed and distutils still being required.

mreyes commented 2 weeks ago

Description:

I'm running SAM CLI, version 1.116.0 on a Mac with python 3.12, already have installed setuptools 60.5.0 as well and yet getting the following error:

Runtime.ImportModuleError: Unable to import module ‘my.module’: No module named 'distutils'

Steps to reproduce:

Trying to run a lambda application locally with python 3.12:

sam local start-api --debug --port 3000

Here is the pip list: Package Version

attrs 21.4.0 awscli 1.33.4 beautifulsoup4 4.12.3 boto3 1.34.122 botocore 1.34.122 bs4 0.0.2 certifi 2020.6.20 cffi 1.16.0 chardet 3.0.4 charset-normalizer 3.3.2 colorama 0.4.6 cryptography 42.0.8 docutils 0.16 idna 2.10 jmespath 1.0.1 jsonpickle 1.0 jsonschema 3.2.0 oauthlib 3.1.0 packaging 24.1 pip 24.0 psycopg2 2.9.9 pyasn1 0.6.0 pycparser 2.22 pyrsistent 0.18.1 pyspnego 0.10.2 python-dateutil 2.9.0.post0 PyYAML 6.0.1 redis 3.5.3 requests 2.24.0 requests-ntlm 1.2.0 requests-oauthlib 1.3.0 rsa 4.7.2 s3transfer 0.10.1 setuptools 60.5.0 six 1.15.0 soupsieve 2.5 urllib3 1.25.9

Observed result:

[ERROR] Runtime.ImportModuleError: Unable to import module 'my.module': No module named 'distutils'

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: MacOS Sonoma Version 14.4.1
sam --version: version 1.116.0
AWS region: us-east-1

# Paste the output of `sam --info` here
/src/layers-static/python/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (2.2.1) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
{
  "version": "1.116.0",
  "system": {
    "python": "3.12.3",
    "os": "macOS-14.4.1-x86_64-i386-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.5",
    "aws_cdk": "Not available",
    "terraform": "0.13.7"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Invalid lambda response received: Invalid API Gateway Response Keys: {'stackTrace', 'errorType', 'errorMessage', 'requestId'} in {'errorMessage': "Unable to import module 'my.module': No module named 'distutils'", 'errorType':
'Runtime.ImportModuleError', 'requestId': '4c2a7d20-7510-4d44-a6f4-c9035f273df7', 'stackTrace': []}

Add --debug flag to command you are running

mndeveci commented 2 weeks ago

Hi there,

It looks like this is coming from your lambda function, can you confirm that you build your function with all its requirements?

In the meantime, if you can share a reproducible, we can test on our side as well.

mreyes commented 2 weeks ago

@mndeveci

In attention to your observation on the build: I have reviewed the build process and resolved all appearing warnings by updating the referred packages now having the following versions:

chardet.  5.2.0
psycopg2-binary.  2.9.9
requests.  2.32.3
urllib3.  2.2.2

This is the build command:

sam build --use-container --build-image docker.artifactory.my.domain.com/sam/build-python3.12 --template sam-templates/my-template.yaml

Here is the requirements.txt file with the following content: 

psycopg2-binary==2.9.9
setuptools==60.5.0

This is the excerpt of the settings in the template file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  my-app
  SAM Template for my-app server-less stack
Globals:
  Function:
    Handler: app.lambda_handler
    Runtime: python3.12
    Layers:
      - !Ref UtilsLayer
      - !Ref StaticLayer
    Timeout: 900
    Environment:
      Variables:

Resources:
  # common modules
  UtilsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: ../layer-adp/python
      CompatibleRuntimes:
        - python3.12
    Metadata:
      BuildMethod: python3.12
  # static modulea
  StaticLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: ../../layer-static/python
      CompatibleRuntimes:
        - python3.12
    Metadata:
      BuildMethod: python3.12
  # function
  TemplatesForUserAPIFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: ../app-src
      Handler: my_api.get_app
      Events:
        GroupsApi:
          Type: Api 
          Properties:
            Path: /app
            Method: GET

As you can see above, I have 2 lambda layers and as the build goes perfect with no errors or warnings and when calling the service after the application server is initialized then the referred error message comes up.

As a side note:

I noticed is that even though I have LOCALLY installed this psycopg2-binary==2.9.9 package, it turns out that when run the build while being network disconnected I get the following error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: psycopg2-binary==2.9.9

Could this indicate that while having locally the package installed it is retrieved from somewhere else?
Once connected back to internet the dependency gets satisfied and the build goes unblemished.

mndeveci commented 1 week ago

Thanks for providing more details.

I've tried to build a python3.12 lambda function with the requirements below;

psycopg2-binary==2.9.9
setuptools==60.5.0

I was able to build it with no issues.

I saw that you are using a custom build image, may I ask why you need it? Can you just run sam build --use-container --template sam-templates/my-template.yaml to see if it is going to succeed?

I noticed is that even though I have LOCALLY installed this psycopg2-binary==2.9.9 package, it turns out that when run the build while being network disconnected I get the following error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: psycopg2-binary==2.9.9

AWS SAM CLI uses PIP cli to manage dependencies. And it downloads them depending on the architecture of your lambda function. It can't use local dependencies all the time since OS or architecture might be different from what you are targeting. If you prefer, you can run your own build process by defining a Makefile, the process is explained here: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/building-custom-runtimes.html

aws / aws-sam-cli