search

aws / aws-lambda-builders

Python library to compile, build & package AWS Lambda functions for several runtimes & framework
Apache License 2.0
334 stars 139 forks source link

Bug: sam build fails to install onnxruntime==1.18.1 #674

Open zhemaituk opened 1 month ago

zhemaituk commented 1 month ago

Description:

onnxruntime==1.18.1 dependency fails to install in container and locally using sam build.

Steps to reproduce:

  1. SAM init hello world example
  2. Runtime: python3.12; Architectures: - arm64
  3. Add to requirements.txt: 
    onnxruntime==1.18.1
  4. sam build --use-container --debug

Observed result:

2024-08-14 11:08:27,171 | Mounting .../hello_world as /tmp/samcli/source:ro,delegated, inside runtime container                                                  
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
...
 Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
pip stdout: b'Collecting onnxruntime==1.18.1 (from -r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading onnxruntime-1.18.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (4.3 kB)\nCollecting coloredlogs (from onnxruntime==1.18.1->-r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\nCollecting flatbuffers (from onnxruntime==1.18.1->-r /tmp/samcli/source/requirements.txt (line 1))\n  Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes)\nCollecting numpy<2.0,>=1.21.6 (from onnxruntime==1.18.1->-r /tmp/samcli/source/requirements.txt (line 1))\n  ...
Full dependency closure: {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), onnxruntime==1.18.1(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
initial compatible: {flatbuffers==24.3.25(wheel), mpmath==1.3.0(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
initial incompatible: {onnxruntime==1.18.1(wheel)}
Downloading missing wheels: {onnxruntime==1.18.1(wheel)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp312 --dest /tmp/samcli/scratch onnxruntime==1.18.1
pip stdout: b''
(from versions: none)\nERROR: No matching distribution found for onnxruntime==1.18.1\n\n[notice] A new release of pip is available: 24.0 -> 24.2\n[notice] To update, run: pip install --upgrade pip\n"
compatible wheels after second download pass: {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Build missing wheels from sdists (C compiling True): set()
compatible after building wheels (no C compiling): {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Build missing wheels from sdists (C compiling False): set()
compatible after building wheels (C compiling): {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Final compatible: {mpmath==1.3.0(wheel), flatbuffers==24.3.25(wheel), packaging==24.1(wheel), sympy==1.13.2(wheel), humanfriendly==10.0(wheel), protobuf==5.27.3(wheel), numpy==1.26.4(wheel), coloredlogs==15.0.1(wheel)}
Final incompatible: {onnxruntime==1.18.1(wheel)}
Final missing wheels: {onnxruntime==1.18.1(wheel)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 71, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/lambda-builders/lib/python3.12/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 265, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {onnxruntime==1.18.1(wheel)}

During handling of the above exception, another exception occurred:

...
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {onnxruntime==1.18.1(wheel)}

During handling of the above exception, another exception occurred:

...
aws_lambda_builders.actions.ActionFailedError: {onnxruntime==1.18.1(wheel)}

During handling of the above exception, another exception occurred:

...
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {onnxruntime==1.18.1(wheel)}

...

Build Failed
...                                                                                                                             
Error: PythonPipBuilder:ResolveDependencies - {onnxruntime==1.18.1(wheel)}

Expected result:

Build Successful

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

The same fails without --use-container as well, BUT only during first execution of sam build, second execution of sam build without cleaning .aws-sam folder is successful.

pip install onnxruntime==1.18.1 by itself works fine in both: container and locally.

OS: macOS 14.6.1 23G93 SAM CLI: version 1.121.0

{
  "version": "1.121.0",
  "system": {
    "python": "3.12.5",
    "os": "macOS-14.6.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "27.1.1",
    "aws_cdk": "Not available",
    "terraform": "1.9.4"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}
hawflau commented 1 month ago

@zhemaituk Thanks for raising the issue.

The package onnxruntime only provides Linux wheels for manylinux_2_27 and manylinux_2_28 and does not provide sdist. But Lambda Builders seems to not support these two manylinux tags. Marking this as a bug.