search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.51k stars 1.17k forks source link

sam build failing to resolve dependencies {simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)} #1730

Closed NTonani closed 4 years ago

NTonani commented 4 years ago

Description

Build is failing to resolve a couple dependencies:
{simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)} I am using pipenv / requirements.txt to manage.

If I run in container, it succeeds. That along with the many suggestions in other issues leads to pip / python mismatch. 

pip --version
pip 19.3.1 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
python --version
Python 3.7.6
sam --version
SAM CLI, version 0.40.0

If I manually run pip download -r /Users/ntonani/workplace/kopa/kopa_serverless/kopa-message-receiver/src/requirements.txt --dest /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 - the failing command - it succeeds. This leads me to believe the pip used by SAM is different.

How do I find out which pip SAM is using?

Observed result

Running PythonPipBuilder:ResolveDependencies
calling pip download -r /Users/ntonani/workplace/kopa/kopa_serverless/kopa-message-receiver/src/requirements.txt --dest /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96
Full dependency closure: {docutils==0.15.2(wheel), chardet==3.0.4(wheel), idna==2.8(wheel), boto3==1.11.3(wheel), python-levenshtein==0.12.0(sdist), certifi==2019.11.28(wheel), setuptools==45.0.0(wheel), twilio==6.35.2(sdist), botocore==1.14.3(wheel), urllib3==1.25.7(wheel), fuzzywuzzy==0.17.0(wheel), simplejson==3.17.0(wheel), requests==2.22.0(wheel), s3transfer==0.3.0(wheel), python-dateutil==2.8.1(wheel), pyjwt==1.7.1(wheel), pytz==2019.3(wheel), six==1.14.0(wheel), jmespath==0.9.4(wheel)}
initial compatible: {pyjwt==1.7.1(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel), botocore==1.14.3(wheel), idna==2.8(wheel), urllib3==1.25.7(wheel), boto3==1.11.3(wheel), s3transfer==0.3.0(wheel), pytz==2019.3(wheel), python-dateutil==2.8.1(wheel), certifi==2019.11.28(wheel), fuzzywuzzy==0.17.0(wheel), six==1.14.0(wheel), jmespath==0.9.4(wheel), setuptools==45.0.0(wheel), docutils==0.15.2(wheel)}
initial incompatible: {simplejson==3.17.0(wheel), twilio==6.35.2(sdist), python-levenshtein==0.12.0(sdist)}
Downloading missing wheels: {twilio==6.35.2(sdist), simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp37m --dest /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 twilio==6.35.2
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp37m --dest /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 simplejson==3.17.0
calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp37m --dest /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 python-levenshtein==0.12.0
compatible wheels after second download pass: {pyjwt==1.7.1(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel), botocore==1.14.3(wheel), idna==2.8(wheel), urllib3==1.25.7(wheel), boto3==1.11.3(wheel), s3transfer==0.3.0(wheel), six==1.14.0(wheel), python-dateutil==2.8.1(wheel), fuzzywuzzy==0.17.0(wheel), jmespath==0.9.4(wheel), pytz==2019.3(wheel), certifi==2019.11.28(wheel), setuptools==45.0.0(wheel), docutils==0.15.2(wheel)}
Build missing wheels from sdists (C compiling True): {twilio==6.35.2(sdist), python-levenshtein==0.12.0(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96/twilio-6.35.2.tar.gz
calling pip wheel --no-deps --wheel-dir /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96/python-Levenshtein-0.12.0.tar.gz
compatible after building wheels (no C compiling): {pyjwt==1.7.1(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel), twilio==6.35.2(wheel), botocore==1.14.3(wheel), idna==2.8(wheel), urllib3==1.25.7(wheel), boto3==1.11.3(wheel), s3transfer==0.3.0(wheel), six==1.14.0(wheel), python-dateutil==2.8.1(wheel), fuzzywuzzy==0.17.0(wheel), jmespath==0.9.4(wheel), pytz==2019.3(wheel), certifi==2019.11.28(wheel), setuptools==45.0.0(wheel), docutils==0.15.2(wheel)}
Build missing wheels from sdists (C compiling False): {python-levenshtein==0.12.0(sdist)}
calling pip wheel --no-deps --wheel-dir /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96 /var/folders/59/dtdk2xg14n13v15zydzkqtwh0000gn/T/tmps99l0k96/python-Levenshtein-0.12.0.tar.gz
compatible after building wheels (C compiling): {pyjwt==1.7.1(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel), twilio==6.35.2(wheel), botocore==1.14.3(wheel), idna==2.8(wheel), urllib3==1.25.7(wheel), boto3==1.11.3(wheel), s3transfer==0.3.0(wheel), six==1.14.0(wheel), python-dateutil==2.8.1(wheel), fuzzywuzzy==0.17.0(wheel), jmespath==0.9.4(wheel), pytz==2019.3(wheel), certifi==2019.11.28(wheel), setuptools==45.0.0(wheel), docutils==0.15.2(wheel)}
Final compatible: {docutils==0.15.2(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel), idna==2.8(wheel), boto3==1.11.3(wheel), s3transfer==0.3.0(wheel), python-dateutil==2.8.1(wheel), certifi==2019.11.28(wheel), setuptools==45.0.0(wheel), pyjwt==1.7.1(wheel), twilio==6.35.2(wheel), botocore==1.14.3(wheel), urllib3==1.25.7(wheel), pytz==2019.3(wheel), fuzzywuzzy==0.17.0(wheel), six==1.14.0(wheel), jmespath==0.9.4(wheel)}
Final incompatible: {simplejson==3.17.0(wheel), python-levenshtein==0.12.0(wheel)}
Final missing wheels: {simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 39, in execute
    requirements_path=self.manifest_path
  File "/usr/local/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 144, in build_dependencies
    requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 213, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflow.py", line 248, in run
    action.execute()
  File "/usr/local/Cellar/aws-sam-cli/0.40.0/libexec/lib/python3.7/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)}

Build Failed
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 23124, 'exitReason': 'BuildError', 'exitCode': 1, 'requestId': '92353e64-20a6-4758-baa2-561816dfc34a', 'installationId': '09d46a47-ea3d-4637-a750-319c111aa184', 'sessionId': 'bcb4867a-958c-45dd-8647-40b19ad9a85b', 'executionEnvironment': 'CLI', 'pyversion': '3.7.6', 'samcliVersion': '0.40.0'}}]}
Telemetry response: 200
Error: PythonPipBuilder:ResolveDependencies - {simplejson==3.17.0(wheel), python-levenshtein==0.12.0(sdist)}

Expected result

sam build succeeds

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: macOS Catalina 10.15.2
  2. sam --version: 0.40.0
sriram-mv commented 4 years ago

The pip that is used is python -m pip and which python is being used is decided by the first python that is encountered on PATH that matches the python runtime version specified on the template.

djm commented 4 years ago

tl;dr pip install wheel, without usage of --use-container.

This fixed it for me and I think you are encountering the same problem, but it's hard for me to say for sure with only having access to the traceback.

I will try and explain..

The failing command is not the pip download you mentioned in the original post, but a later pip wheel command. aws-sam-cli relies on aws-lambda-builders to handle the pip download part of SAM.

It attempts to get the platform-specific binary wheels when it can; but not all packages are shipped to PyPi with those wheels. So in those cases, it seems it tries to build the wheel from the sdist, and for that it requires the wheel package.. and when that does not exist - you get this MissingDependencies error.

I'm not sure on the reasoning behind not having wheel as a dependency (or an optional dependency when using Python SAM functions); it seems like that would save a lot of people a great amount of time with these cryptic errors.

If you take a look at aws-lambda-builders here it is explicitly ignoring the fact that pip wheel might fail; I don't know about the wider program, perhaps there is a good reason, but this doesn't seem wise to me.

HTH.

sriram-mv commented 4 years ago

Going to close this issue. Sam build --use-container makes sure that the wheels that are built are directly usable in a lambda environment. Looks like levenstien for example depends on certain C bindings.

What @djm points out is also true, that you need the wheel package to be able to build wheels. (Need better documentation on this).

NTonani commented 4 years ago

@sriram-mv Can you explain further? Not sure what '...makes sure that the wheels that are built are directly usable in a lambda environment' means.

I have other dependencies that depend on wheel builds so the issue is not missing the wheel package.

I'm not able to use --use-container any longer as I would require a custom base Docker image, which isn't supported.

abelash commented 4 years ago

tl;dr pip install wheel, without usage of --use-container.

@djm Could you explain more about your approach? I've got the same problem trying run local lambda function from PyCharm with AWS Toolkit plugin. I've tried both options(with and without --use-container) running sam build Function and got error:
PythonPipBuilder:ResolveDependencies - {fbprophet==0.6(sdist), ephem==3.7.7.1(wheel)}

jmcgrath207 commented 3 years ago

This happened to me because my system interrupter 3.9 was not compatible with my sam 3.8 python project.

Removing the sam system install and replace it with the pip version in my requirement files made it work again.

https://pypi.org/project/aws-sam-cli/1.20.0/