search

aws / aws-lambda-builders

Python library to compile, build & package AWS Lambda functions for several runtimes & framework
Apache License 2.0
338 stars 139 forks source link

python arm build issue still failing in some cases #348

Open blathersbugs opened 2 years ago

blathersbugs commented 2 years ago

Description:

python lambda function builds using the sam cli (which calls this library) are still failing for certain dependencies

Steps to reproduce the issue:

  1. setup python project with sam cli
  2. add py7zr to the requirements.txt of a function
  3. run command sam build -u

Observed result: build fails on resolve dependencies Expected result: build completes Additional environment details (Ex: Windows, Mac, Amazon Linux etc) aws codebuild image aws/codebuild/amazonlinux2-x86_64-standard:3.0 buildspec.yaml: version: 0.2 env: secrets-manager: DOCKER_PASSWORD: *** DOCKER_USER: ** phases: install: runtime-versions: python: 3.9 build: commands:

sam build -u --debug output near failure:

2022-03-22 18:35:43,680 | Building codeuri: /codebuild/output/src531839338/src/lambda/**** runtime: python3.9 metadata: {} architecture: arm64 functions: ['*'] 2022-03-22 18:35:43,680 | Building to following folder /codebuild/output/src531839338/src/.aws-sam/build

Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image...... 2022-03-22 18:35:44,617 | Mounting /codebuild/output/src531839338/src/lambda/** as /tmp/samcli/source:ro,delegated inside runtime container Using the request object from command line argument Loading workflow module 'aws_lambda_builders.workflows' Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)' Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)' Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)' Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' Running workflow 'PythonPipBuilder' Running PythonPipBuilder:ResolveDependencies calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i Full dependency closure: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(sdist), texttable==1.6.4(wheel), zipfile-deflate64==0.0.0(sdist), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)} initial compatible: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)} initial incompatible: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)} Downloading missing wheels: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)} calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch pycryptodomex==3.14.1 calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch zipfile-deflate64==0.0.0 compatible wheels after second download pass: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)} Build missing wheels from sdists (C compiling True): {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)} calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycryptodomex-3.14.1.tar.gz calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz compatible after building wheels (no C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)} Build missing wheels from sdists (C compiling False): {zipfile-deflate64==0.0.0(sdist)} calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz compatible after building wheels (C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)} Final compatible: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)} Final incompatible: set() Final missing wheels: {zipfile-deflate64==0.0.0(sdist)} PythonPipBuilder:ResolveDependencies failed Traceback (most recent call last): File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute package_builder.build_dependencies( File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages raise MissingDependencyError(packages_without_wheels) aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run action.execute() File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute raise ActionFailedError(str(ex)) aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)} Builder workflow failed Traceback (most recent call last): File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute package_builder.build_dependencies( File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages raise MissingDependencyError(packages_without_wheels) aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run action.execute() File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute raise ActionFailedError(str(ex)) aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/main.py", line 118, in main builder.build( File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 164, in build return workflow.run() File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 96, in wrapper func(self, *args, **kwargs) File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 309, in run raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex)) aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)} 2022-03-22 18:43:02,875 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}"}}

Build Failed 2022-03-22 18:43:03,179 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '64785e7d-a17c-4f7f-a347-aca7b7adc804', 'installationId': '75a7108d-f9a0-4aa3-ad9a-b637b8ea0647', 'sessionId': 'c7ec421f-a00b-47f1-b1e6-b34189e99768', 'executionEnvironment': 'AWSCodeBuild', 'ci': True, 'pyversion': '3.9.5', 'samcliVersion': '1.41.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 667631, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]} 2022-03-22 18:43:03,217 | Telemetry response: 200 Error: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}

[Container] 2022/03/22 18:43:03 Command did not exit successfully sam build -u --debug exit status 1 [Container] 2022/03/22 18:43:03 Phase complete: BUILD State: FAILED [Container] 2022/03/22 18:43:03 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: sam build -u --debug. Reason: exit status 1 [Container] 2022/03/22 18:43:03 Entering phase POST_BUILD [Container] 2022/03/22 18:43:03 Phase complete: POST_BUILD State: SUCCEEDED [Container] 2022/03/22 18:43:03 Phase context status code: Message: [Container] 2022/03/22 18:43:03 Expanding base directory path: . [Container] 2022/03/22 18:43:03 Assembling file list [Container] 2022/03/22 18:43:03 Expanding . [Container] 2022/03/22 18:43:03 Expanding file paths for base directory . [Container] 2022/03/22 18:43:03 Assembling file list [Container] 2022/03/22 18:43:03 Expanding template-out.yaml [Container] 2022/03/22 18:43:03 Skipping invalid file path template-out.yaml [Container] 2022/03/22 18:43:03 Phase complete: UPLOAD_ARTIFACTS State: FAILED [Container] 2022/03/22 18:43:03 Phase context status code: CLIENT_ERROR Message: no matching artifact paths found

mildaniel commented 2 years ago

Thanks for reporting the issue, we will be investigating it further. Please watch this channel for more updates, and feel free to reach out.

blathersbugs commented 2 years ago

another function with the following requirements:

requests python_jwt

plus a layer with this:

dynamodb-encryption-sdk

that was failing to build before is now building successfully, if that helps. (the layer was always succeeding) none of the versions is pinned, so unfortunately that makes more variables to deal with.

mndeveci commented 2 years ago

@blathersbugs thanks for raising this issue.

I've tried with following dependencies with python3.9 runtime, and I confirm that it works if I run sam build -u;

requests
python_jwt
dynamodb-encryption-sdk

Can you provide more details for these ones (python version for instance) to test it further?

I also tried with py7zr as dependency, but that one fails with following message;

Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}

When I check PyPI, I can't find the 0.0.0 version of zipfile-deflate64. Do you know where that dependency & version comes from?

blathersbugs commented 2 years ago

hi @mndeveci 

requests
python_jwt
dynamodb-encryption-sdk

work as expected, so there is nothing to reproduce there. I don't know where where it gets the 0.0.0 version. If you look through the output, you can see it builds the correct wheel here compatible after building wheels (C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)} but because of the bogus 0.0.0 version, it believes it does not have all dependencies and the build fails. I was using python 3.9.

if you download the py7zr source files it lists the following as its depedencies: dependencies = [ "texttable", "pycryptodomex>=3.6.6", 'importlib_metadata;python_version<"3.8"', 'brotli>=1.0.9;platform_python_implementation=="CPython"', 'brotlicffi>=1.0.9.2;platform_python_implementation=="PyPy"', "pyzstd>=0.14.4", "pyppmd>=0.18.1,<0.19.0", 'pybcj>=0.5.0;platform_python_implementation=="CPython"', "multivolumefile>=0.2.3", "zipfile-deflate64>=0.2.0", ] so I would say there is an issue in the dependency resolution

blathersbugs commented 2 years ago

hi @mndeveci @mildaniel any updates? is there any more information you need from me?

mndeveci commented 2 years ago

Hi @blathersbugs ,

I've retired with following function definition;

 HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - arm64

with following requirements.txt file;

py7zr
requests
python_jwt
dynamodb-encryption-sdk

when I run sam build -u I got my function built successfully.

I remember this was failing before so please let me know if I am missing something here. Otherwise I would assume the library have fixed the issue with their dependencies that we discussed earlier.

Thanks

blathersbugs commented 2 years ago

hi @mndeveci I can report that the build is now succeeding for me as well, I see no reason not to close this issue.

blathersbugs commented 2 years ago

@mndeveci this has started occurring again for me, it did work for a while and now it isn't. Installing py7zr via pip works fine on a t4g amazon linux 2 instance if that helps, I tried that several times and it worked consistently.

It failed on something different this time, here is the logs:

2022-07-14 18:26:00,608 \| Building codeuri: /codebuild/output/src567373820/src/lambda/hp-png-cap-unzip runtime: python3.9 metadata: {} architecture: arm64 functions: unzip -- 894 | 2022-07-14 18:26:00,609 \| Building to following folder /codebuild/output/src567373820/src/.aws-sam/build/unzip 895 |   896 | Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image...... 897 | 2022-07-14 18:26:01,536 \| Mounting /codebuild/output/src567373820/src/lambda/hp-png-cap-unzip as /tmp/samcli/source:ro,delegated inside runtime container 898 | Using the request object from command line argument 899 | Loading workflow module 'aws_lambda_builders.workflows' 900 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' 901 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' 902 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 903 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' 904 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' 905 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' 906 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' 907 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)' 908 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)' 909 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 910 | Running workflow 'PythonPipBuilder' 911 | Running PythonPipBuilder:ResolveDependencies 912 | calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i 913 | Full dependency closure: {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), psutil==5.9.1(sdist), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), brotli==1.0.9(wheel), pycryptodomex==3.15.0(wheel), unknown==0.0.0(sdist), multivolumefile==0.2.3(wheel)} 914 | initial compatible: {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), brotli==1.0.9(wheel), pycryptodomex==3.15.0(wheel), multivolumefile==0.2.3(wheel)} 915 | initial incompatible: {unknown==0.0.0(sdist), psutil==5.9.1(sdist)} 916 | Downloading missing wheels: {unknown==0.0.0(sdist), psutil==5.9.1(sdist)} 917 | calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch unknown==0.0.0 918 | calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch psutil==5.9.1 919 | compatible wheels after second download pass: {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), pycryptodomex==3.15.0(wheel), brotli==1.0.9(wheel), multivolumefile==0.2.3(wheel)} 920 | Build missing wheels from sdists (C compiling True): {unknown==0.0.0(sdist), psutil==5.9.1(sdist)} 921 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/inflate64-0.1.4.tar.gz 922 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/psutil-5.9.1.tar.gz 923 | compatible after building wheels (no C compiling): {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), psutil==5.9.1(wheel), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), pycryptodomex==3.15.0(wheel), brotli==1.0.9(wheel), multivolumefile==0.2.3(wheel), inflate64==0.1.4(wheel)} 924 | Build missing wheels from sdists (C compiling False): {unknown==0.0.0(sdist)} 925 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/inflate64-0.1.4.tar.gz 926 | compatible after building wheels (C compiling): {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), psutil==5.9.1(wheel), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), pycryptodomex==3.15.0(wheel), brotli==1.0.9(wheel), multivolumefile==0.2.3(wheel), inflate64==0.1.4(wheel)} 927 | Final compatible: {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), psutil==5.9.1(wheel), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), pycryptodomex==3.15.0(wheel), brotli==1.0.9(wheel), multivolumefile==0.2.3(wheel), inflate64==0.1.4(wheel)} 928 | Final incompatible: set() 929 | Final missing wheels: {unknown==0.0.0(sdist)} 930 | PythonPipBuilder:ResolveDependencies failed 931 | Traceback (most recent call last): 932 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute 933 | package_builder.build_dependencies( 934 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies 935 | self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) 936 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages 937 | raise MissingDependencyError(packages_without_wheels) 938 | aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {unknown==0.0.0(sdist)} 939 |   940 | During handling of the above exception, another exception occurred: 941 |   942 | Traceback (most recent call last): 943 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 301, in run 944 | action.execute() 945 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute 946 | raise ActionFailedError(str(ex)) 947 | aws_lambda_builders.actions.ActionFailedError: {unknown==0.0.0(sdist)} 948 | Builder workflow failed 949 | Traceback (most recent call last): 950 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute 951 | package_builder.build_dependencies( 952 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies 953 | self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) 954 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages 955 | raise MissingDependencyError(packages_without_wheels) 956 | aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {unknown==0.0.0(sdist)} 957 |   958 | During handling of the above exception, another exception occurred: 959 |   960 | Traceback (most recent call last): 961 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 301, in run 962 | action.execute() 963 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute 964 | raise ActionFailedError(str(ex)) 965 | aws_lambda_builders.actions.ActionFailedError: {unknown==0.0.0(sdist)} 966 |   967 | During handling of the above exception, another exception occurred: 968 |   969 | Traceback (most recent call last): 970 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/__main__.py", line 118, in main 971 | builder.build( 972 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 164, in build 973 | return workflow.run() 974 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 95, in wrapper 975 | func(self, *args, **kwargs) 976 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 308, in run 977 | raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex)) 978 | aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)} 979 | 2022-07-14 18:31:55,152 \| Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)}"}} 980 |   981 | Build Failed 982 | 2022-07-14 18:31:55,357 \| Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '58da160e-55f9-4380-9564-94df38ffbe73', 'installationId': '94875c77-9f56-485f-975b-b6e447bac83e', 'sessionId': '15165d6b-b6f4-4d5d-b7fd-f29cf0be3b92', 'executionEnvironment': 'AWSCodeBuild', 'ci': True, 'pyversion': '3.9.12', 'samcliVersion': '1.53.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 637988, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]} 983 | 2022-07-14 18:31:55,475 \| Telemetry response: 200 984 | Error: PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)}
jfuss commented 2 years ago

@blathersbugs This is a dependency problem. From the output: Full dependency closure: {pyppmd==0.18.3(wheel), pyzstd==0.15.2(wheel), psutil==5.9.1(sdist), pybcj==0.6.1(wheel), py7zr==0.19.0(wheel), texttable==1.6.4(wheel), brotli==1.0.9(wheel), pycryptodomex==3.15.0(wheel), unknown==0.0.0(sdist), multivolumefile==0.2.3(wheel)}. You can see the unknown package being there which cannot be built. The package on PyPi looks strange: https://pypi.org/project/UNKNOWN/. You will need to figure out what packages are pulling this in from your dependency tree.

blathersbugs commented 2 years ago

@jfuss here is the output from installing py7zr on a t4g amazon linux 2 instance with pip: (env) sh-4.2$ python -m pip --no-cache-dir install py7zr Collecting py7zr Downloading py7zr-0.19.0-py3-none-any.whl (65 kB) |████████████████████████████████| 65 kB 398 kB/s Collecting pyppmd<0.19.0,>=0.18.1 Downloading pyppmd-0.18.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (136 kB) |████████████████████████████████| 136 kB 853 kB/s Collecting brotli>=1.0.9; platform_python_implementation == "CPython" Downloading Brotli-1.0.9-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB) |████████████████████████████████| 2.6 MB 1.7 MB/s Collecting inflate64>=0.1.2; python_version > "3.6" Downloading inflate64-0.1.4.tar.gz (148 kB) |████████████████████████████████| 148 kB 25.9 MB/s Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing wheel metadata ... done Collecting psutil Downloading psutil-5.9.1.tar.gz (479 kB) |████████████████████████████████| 479 kB 14.2 MB/s Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Collecting multivolumefile>=0.2.3 Downloading multivolumefile-0.2.3-py3-none-any.whl (17 kB) Collecting pycryptodomex>=3.6.6 Downloading pycryptodomex-3.15.0-cp35-abi3-manylinux2014_aarch64.whl (1.6 MB) |████████████████████████████████| 1.6 MB 10.4 MB/s Collecting pybcj>=0.6.0 Downloading pybcj-0.6.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (51 kB) |████████████████████████████████| 51 kB 17.3 MB/s Collecting texttable Downloading texttable-1.6.4-py2.py3-none-any.whl (10 kB) Collecting pyzstd>=0.14.4 Downloading pyzstd-0.15.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.1 MB) |████████████████████████████████| 2.1 MB 4.0 MB/s Building wheels for collected packages: inflate64, psutil Building wheel for inflate64 (PEP 517) ... done Created wheel for inflate64: filename=inflate64-0.1.4-cp38-cp38-linux_aarch64.whl size=80513 sha256=00260a6e001ee283050e5a5f8284a148053edfd562f8f093b2a52802306df21a Stored in directory: /tmp/pip-ephem-wheel-cache-ornyh953/wheels/a8/a4/90/870dd7cc09c1c8fa60c4c38f5a9e6e975afdb6e28dfbc93f00 Building wheel for psutil (PEP 517) ... done Created wheel for psutil: filename=psutil-5.9.1-cp38-cp38-linux_aarch64.whl size=298003 sha256=cd5adf56b9d91f78895180a0ad2f3cdaed6e2aca996d61cb5a771074a1d74983 Stored in directory: /tmp/pip-ephem-wheel-cache-ornyh953/wheels/69/3f/b4/d3ed6919124e311eab9e7b9a264f31fd43ae8e7241d4611228 Successfully built inflate64 psutil Installing collected packages: pyppmd, brotli, inflate64, psutil, multivolumefile, pycryptodomex, pybcj, texttable, pyzstd, py7zr Successfully installed brotli-1.0.9 inflate64-0.1.4 multivolumefile-0.2.3 psutil-5.9.1 py7zr-0.19.0 pybcj-0.6.1 pycryptodomex-3.15.0 pyppmd-0.18.3 pyzstd-0.15.2 texttable-1.6.4

So I don't think it is a dependency at all, I think there is some sort of issue with the dependency resolution used by aws-lambda-builders. I will try to narrow down which of these packages is causing it if I can.

blathersbugs commented 2 years ago

hi @mndeveci @jfuss it fails with inflate64, here is the build output:

Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... -- 193 | 2022-07-15 20:43:34,641 \| Mounting /codebuild/output/src341811617/src/lambda/helloworld as /tmp/samcli/source:ro,delegated inside runtime container 194 | Using the request object from command line argument 195 | Loading workflow module 'aws_lambda_builders.workflows' 196 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' 197 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' 198 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 199 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' 200 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' 201 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' 202 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' 203 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)' 204 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)' 205 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 206 | Running workflow 'PythonPipBuilder' 207 | Running PythonPipBuilder:ResolveDependencies 208 | calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i 209 | Full dependency closure: {unknown==0.0.0(sdist)} 210 | initial compatible: set() 211 | initial incompatible: {unknown==0.0.0(sdist)} 212 | Downloading missing wheels: {unknown==0.0.0(sdist)} 213 | calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch unknown==0.0.0 214 | compatible wheels after second download pass: set() 215 | Build missing wheels from sdists (C compiling True): {unknown==0.0.0(sdist)} 216 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/inflate64-0.1.4.tar.gz 217 | compatible after building wheels (no C compiling): {inflate64==0.1.4(wheel)} 218 | Build missing wheels from sdists (C compiling False): {unknown==0.0.0(sdist)} 219 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/inflate64-0.1.4.tar.gz 220 | compatible after building wheels (C compiling): {inflate64==0.1.4(wheel)} 221 | Final compatible: {inflate64==0.1.4(wheel)} 222 | Final incompatible: set() 223 | Final missing wheels: {unknown==0.0.0(sdist)} 224 | PythonPipBuilder:ResolveDependencies failed 225 | Traceback (most recent call last): 226 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute 227 | package_builder.build_dependencies( 228 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies 229 | self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) 230 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages 231 | raise MissingDependencyError(packages_without_wheels) 232 | aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {unknown==0.0.0(sdist)} 233 |   234 | During handling of the above exception, another exception occurred: 235 |   236 | Traceback (most recent call last): 237 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 301, in run 238 | action.execute() 239 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute 240 | raise ActionFailedError(str(ex)) 241 | aws_lambda_builders.actions.ActionFailedError: {unknown==0.0.0(sdist)} 242 | Builder workflow failed 243 | Traceback (most recent call last): 244 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute 245 | package_builder.build_dependencies( 246 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies 247 | self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) 248 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages 249 | raise MissingDependencyError(packages_without_wheels) 250 | aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {unknown==0.0.0(sdist)} 251 |   252 | During handling of the above exception, another exception occurred: 253 |   254 | Traceback (most recent call last): 255 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 301, in run 256 | action.execute() 257 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute 258 | raise ActionFailedError(str(ex)) 259 | aws_lambda_builders.actions.ActionFailedError: {unknown==0.0.0(sdist)} 260 |   261 | During handling of the above exception, another exception occurred: 262 |   263 | Traceback (most recent call last): 264 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/__main__.py", line 118, in main 265 | builder.build( 266 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 164, in build 267 | return workflow.run() 268 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 95, in wrapper 269 | func(self, *args, **kwargs) 270 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 308, in run 271 | raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex)) 272 | aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)} 273 | 2022-07-15 20:47:47,499 \| Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)}"}}

let me check if any of the other dependencies cause this as well

EDIT: all other dependencies succeed in building

I downloaded the inflate64 source code from pypi and the word "sdist" only appears in a readme file, it isn't in any of the code or config files.

jfuss commented 2 years ago

@blathersbugs If you take a look at https://github.com/aws/aws-lambda-builders/blob/64f478f6b4c4ef3482222bbd5b1588402cab5635/aws_lambda_builders/workflows/python_pip/packager.py#L359 (which is where Full dependency closure: {unknown==0.0.0(sdist)} gets logged from), you will see we first download the dependencies using pip (command 208 | calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i from your log). So somewhere in your dependency closure, pip is getting unknown and therefore we are trying to build it.

This looks to me like an issue with your system or dependency closure.

blathersbugs commented 2 years ago

@jfuss the system is aws codebuild, this image: aws/codebuild/amazonlinux2-x86_64-standard:3.0, let me try with a native arm64 codebuild image

jfuss commented 2 years ago

@blathersbugs Can you provide output from --debug. I poking through the code, and the only thing I can see if it pypi is returning that back to us somehow and therefore hoping debug logs have a little more info.

The other thing you can try is doing the pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i command with your requirements.txt. That might give more info in regards to what pip is returning.

blathersbugs commented 2 years ago

@jfuss all the logs I pasted were --debug logs., unless you mean from the t4g server where it succeeded. I retried it with this codebuild image: aws/codebuild/amazonlinux2-aarch64-standard:2.0 and it still failed. i will try running the command you suggested on my t4g server. EDIT: even with -vvv, the term "sdist" doesn't appear anywhere in the output of that command

jfuss commented 2 years ago

even with -vvv, the term "sdist" doesn't appear anywhere in the output of that command

We output "sdist" because we couldn't find a .whl for the package.

blathersbugs commented 2 years ago

@jfuss am I correct in assuming that outputting "sdist" is supposed to trigger building the package from source? If that is so, then what I think is happening is it is finding the package sdist https://pypi.org/project/sdist/ that someone uploaded on pypi and attempting to install that package instead. This then fails because that package does not have a compatible version. (from sdist PKG-INFO):

Metadata-Version: 1.0
Name: sdist
Version: 0.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN

sdist-0.0.0.tar.gz

or at least I think this might have something to do with what is happening

blathersbugs commented 1 year ago

@jfuss @mndeveci this is still failing for inflate64==0.1.4 it succeeds for newer versions because there is now a wheel available. here is the output from the failing build:

Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image.......................................................................................................................................................................................................................................................................................................................................................................................................................... -- 267 | 2023-02-10 18:07:34,019 \| Mounting /codebuild/output/src174021131/src/lambda/helloworld as /tmp/samcli/source:ro,delegated inside runtime container 268 | Using the request object from command line argument 269 | Loading workflow module 'aws_lambda_builders.workflows' 270 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' 271 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' 272 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 273 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' 274 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' 275 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' 276 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' 277 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)' 278 | Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)' 279 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 280 | Running workflow 'PythonPipBuilder' 281 | Running PythonPipBuilder:ResolveDependencies 282 | calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i 283 | Full dependency closure: {unknown==0.0.0(sdist)} 284 | initial compatible: set() 285 | initial incompatible: {unknown==0.0.0(sdist)} 286 | Downloading missing wheels: {unknown==0.0.0(sdist)} 287 | calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch unknown==0.0.0 288 | compatible wheels after second download pass: set() 289 | Build missing wheels from sdists (C compiling True): {unknown==0.0.0(sdist)} 290 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/inflate64-0.1.4.tar.gz 291 | compatible after building wheels (no C compiling): {inflate64==0.1.4(wheel)} 292 | Build missing wheels from sdists (C compiling False): {unknown==0.0.0(sdist)} 293 | calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/inflate64-0.1.4.tar.gz 294 | compatible after building wheels (C compiling): {inflate64==0.1.4(wheel)} 295 | Final compatible: {inflate64==0.1.4(wheel)} 296 | Final incompatible: set() 297 | Final missing wheels: {unknown==0.0.0(sdist)} 298 | PythonPipBuilder:ResolveDependencies failed 299 | Traceback (most recent call last): 300 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute 301 | package_builder.build_dependencies( 302 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies 303 | self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) 304 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 259, in build_site_packages 305 | raise MissingDependencyError(packages_without_wheels) 306 | aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {unknown==0.0.0(sdist)} 307 |   308 | During handling of the above exception, another exception occurred: 309 |   310 | Traceback (most recent call last): 311 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 372, in run 312 | action.execute() 313 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute 314 | raise ActionFailedError(str(ex)) 315 | aws_lambda_builders.actions.ActionFailedError: {unknown==0.0.0(sdist)} 316 | Builder workflow failed 317 | Traceback (most recent call last): 318 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute 319 | package_builder.build_dependencies( 320 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies 321 | self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) 322 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 259, in build_site_packages 323 | raise MissingDependencyError(packages_without_wheels) 324 | aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {unknown==0.0.0(sdist)} 325 |   326 | During handling of the above exception, another exception occurred: 327 |   328 | Traceback (most recent call last): 329 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 372, in run 330 | action.execute() 331 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute 332 | raise ActionFailedError(str(ex)) 333 | aws_lambda_builders.actions.ActionFailedError: {unknown==0.0.0(sdist)} 334 |   335 | During handling of the above exception, another exception occurred: 336 |   337 | Traceback (most recent call last): 338 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/__main__.py", line 118, in main 339 | builder.build( 340 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 171, in build 341 | return workflow.run() 342 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 118, in wrapper 343 | func(self, *args, **kwargs) 344 | File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 379, in run 345 | raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex)) 346 | aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)} 347 | 2023-02-10 18:11:31,271 \| Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {unknown==0.0.0(sdist)}"}} 348 |   349 | Build Failed

it looks like that sdist package on pypi is the issue, at least to me with my limited knowledge of python packaging. it seems to have successfully build inflate64, but it somehow added a dependecy on sdist at an UNKNOWN version.

jfuss commented 1 year ago

@blathersbugs I am not really sure where unknown is coming from. Could you provide the requirements.txt you are using that is causing this failure? I am hoping with the same dependencies, I can repro the issue locally and be able to step through debug.

blathersbugs commented 1 year ago

requirements.txt @jfuss here you go, note that only inflate64 is uncommented, this is the only item needed to trigger the issue. There is also an arm wheel for newer versions, so the pinned version is required to trigger the issue.

jfuss commented 1 year ago

@blathersbugs Got it. Will try to build locally with that to try and repro. Will comment back with more details once I have something

jfuss commented 1 year ago

@blathersbugs I tried locally with x86, I know that is not the same (you are wanting to build for arm) but not seeing the same behavior.

I starting poking deeper into pip because unknown seems to be coming back from https://github.com/aws/aws-lambda-builders/blob/8ad50c31cd8663630df0ee2625a11543e991d1fd/aws_lambda_builders/workflows/python_pip/packager.py#L356. My guess right now, is pip is finding something that it can't resolve as a dependency and returning unknown. SAM CLI/Lambda Builders then think that is a package it needs to handle. I still need to understand deeper but that is how it is looking by some comments in the pip codebase.

You mentioned above that newer versions don't have this issue, is there any reason you can't migrate to a newer version?

jfuss commented 1 year ago

More "googling" seems to suggest an outdated version of setuptools. https://github.com/pypa/setuptools/issues/3511 has more details on what is going on under the hood.

I think failing for when we see unknown dependency might be the right thing, as it doesn't seem like something we can recover from. You might be able to pin setuptools to 63.0.0 (was suggested to work in the above issue) to work around this but I think the better thing is to just upgrade your dependency, since there are supported whl and one of the reasons this is happening is SAM CLI is trying to build the dependency from source.

blathersbugs commented 1 year ago

@jfuss There is nothing stopping me from updating the dependency at this point, the problem is that this issue might still be happening for someone else with another arm package that lacks a wheel, unless we are certain it is only this package.
If you want me to, I can try to find another arm package without a wheel to determine the scope of the problem. If it is just this one package, I am fine with closing the issue. I was building on x86 with a target architecture of arm. You need to run this command to make that work: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes The reason to do this is that while this allows building for arm on x86, there doesn't seem to be a way to build for x86 on arm. Also, when I tried this with a native arm codebuild server, it still failed, although that was a while ago. note that it only fails using the sam cli on codebuild. I cannot verify if sam cli works locally right now since my work computer doesn't play nice with docker. If you need me to, I can try it later on my home computer.

jfuss commented 1 year ago

@blathersbugs I wasn't trying to suggest to close. I think this or something targeting the "unknown" dependency needs to be in the repo until we fully fix it or have better error handling.

I was just trying to give details on what is going on (or at least what I think is). I wanted to make sure you are not blocked by this first, which is why I was bringing up the upgrading.

I doubt this is only this package, honestly but I don't think this is common either.

moelasmar commented 1 year ago

I am able to reproduce this issue.

moelasmar commented 1 year ago

I believe that the source of this issue is the legacy arm64 platforms are missed here

https://github.com/aws/aws-lambda-builders/blob/8305159c383f286937ff8eb851fc2ba92faa6e29/aws_lambda_builders/workflows/python_pip/packager.py#LL191-L195