Open fniko opened 4 years ago
I solved this using sam build --use-container
, however this works only when template.yml file is present.
Using the graphical form does not allow to insert --use-container and it fails, so I do not count this as a solution, but workaround instead...
// Also since I am unable to build app locally in my IDE, I am unable to run/debug it, so it makes no sense to use this workaround.
Are you using AWS Toolkit for Pycharm?
I'm seeing something similar:
$ sam build
Building resource 'PaymentsDataStreamFunction'
Running PythonPipBuilder:ResolveDependencies
Build Failed
Error: PythonPipBuilder:ResolveDependencies - {pyyaml==5.3.1(sdist), pyrsistent==0.15.7(sdist)}
Both pyyaml & pyrsistent are inside my requirements.txt file, which have been installed inside a virtualenv.
Do you have wheel installed? if not sam build --use-container
may be the best bet.
wheel is installed. --use-container
also fails with a similar error but with different deps. I tried creating a brand new project which worked fine. So, it's probably something within my project that is messed up.
@c2tarun yes, I am using this plugin
Same here - Used another dep that depends on jsonschema
and bumped into this. Can't use --use-container
as I have a shared dep two directories up. For that to work I'd need to copy the wheel to every service I have, or have a way to mount a volume during sam build
.
Thanks for opening the issue, I'll spend a couple of days rewriting the lib and build a wheel from that instead
I'm getting same issue when I use sam build --use-container (Windows, Python 3.6, SAM CLI version 0.47.0): Build Failed Error: PythonPipBuilder:ResolveDependencies - {tensorflow==1.15.0(wheel), grpcio==1.28.1(wheel)}
Same. It seems like it fails to build anything with tensorflow or better yet grpcio
My current workaround is to manually add grpcio
to the requirements.txt
file, but fixed to an older version.
grpcio==1.27.2
It seems to fail with version 1.28.
I'm seeing a similar error with the following in requirements.txt
pyorc==0.2.0
I am running:
sam build --use-container
Output: Starting Build inside a container Building resource 'CdcFunction'
Fetching lambci/lambda:build-python3.7 Docker container image...... Mounting /path/to/my/app as /tmp/samcli/source:ro,delegated inside runtime container
Build Failed Running PythonPipBuilder:ResolveDependencies Error: PythonPipBuilder:ResolveDependencies - {pyorc==0.2.0(wheel)}
wheel is installed. I'm getting the same error when I add psycopg2 to the requirements. Here is my template.yaml for reference
Globals:
Function:
Timeout: 180
Resources:
CdcFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: lambda/
Handler: lambda.lambda_handler
Runtime: python3.7
+1
+1
Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.16.0(sdist), pyyaml==5.1.2(sdist)}
If your dependencies have C bindings involved, building within the container is the way forward in those cases. Toolkits have an explicit checkbox that can be enabled to allow for all builds to be within a build container.
Closing this issue, please re-open if you feel this doesn't answer your original question.
got the following error started appearing during sam build
Error: PythonPipBuilder:ResolveDependencies - {protobuf==3.17.1(wheel)}
SAM CLI, version 1.20.0 Python 3.8.7 pip 21.1.2
--use-container
doesn't solve the problem.
UnsatisfiedLinkError: com.amazonaws.kinesisvideo.producer.jni.NativeKinesisVideoProducerJni.getNativeLibraryVersion()Ljava/lang/String I am following this workspace https://github.com/backdoorcodr/amazon-kinesis-video-streams-producer-sdk-java Expectation is to run kinesis video stream using webcam in java. Please help
I was running into Error: PythonPipBuilder:ResolveDependencies - {cytoolz==0.11.0(wheel)}
with sam 0.45.0. Upgrading sam to latest (1.27.2) solved it
Build Failed
Error: PythonPipBuilder:ResolveDependencies - {pillow==8.3.1(wheel)}
Solved updating SAM CLI from version 1.22.0 to version 1.29.0
I'm having the same issue
sam build --use-container --template ./my_template.yaml --debug
produces
Error: PythonPipBuilder:ResolveDependencies - Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-download-ufgf7ow5/fastparquet/
However, installing the requirements.txt with pip in the host machine produces no error.
I also tried to update my sam version from 1.27.2 -> 1.31.0 and I get the same error.
I have the same issue. Tried installing wheel, using --use-container, updating sam to 1.31. None is helpful.
The most weird thing is it was working last week, but suddenly broke today. Nothing has changed at all!
Reopening since this still seems to be an issue. Potentially related: https://github.com/aws/aws-lambda-builders/issues/280
+1
Error: PythonPipBuilder:ResolveDependencies - {olefile==0.46(sdist), future==0.18.2(sdist), wrapt==1.12.1(sdist)}
I was having a similar problem just running from the command line with SAM cli 1.15. A little behind, I know!
Here's what I had before using a python pipenv virtual environment.
OS: macOS Monterey 12.0.1 SAM: SAM CLI, version 1.15.0 P: Python 3.8.7 IDE: 1.61.2 AWS: aws-cli/2.1.21 Python/3.7.4 Darwin/21.1.0 exe/x86_64 prompt/off used wheel: wheel 0.37.0 used pip:21.3.1
I upgraded to 1.34 and build seems to work.
Thanks @ryanaklein ! Unfortunately in my case it's already version 1.34.1, though the other versions are different (for instance, python 3.9).
My environment:
I am building this project: https://github.com/jojo786/awswhatsnew, which works perfectly locally, but not with sam.
If i do a sam build
without a requirements.txt, its successful. But with this requirements.txt file of:
feedparser
requests
python-twitter
aws_lambda_powertools
I get this when doing a sam build
Running PythonPipBuilder:ResolveDependencies
Build Failed
Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.3(wheel)}
These are the things I have tried unsuccessfully:
--use-container
Linked issues:
This fixed the ResolveDependencies error for me
pip install wheel
I am also seeing a problem. I can deploy to AWS without anything in my requirements file, but when I add boto3 (for instance to my requirements file I get the following error)
Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
My requirements.txt looks like this...
boto3==1.18.0 botocore==1.21.65 jmespath==0.10.0 python-dateutil==2.8.2 s3transfer==0.5.0 six==1.16.0 urllib3==1.26.7
I've tried upgrading pip and wheel but this has not helped.
This definitely seems to be some sort of pip
issue around string encoding, lots of results like this: https://stackoverflow.com/questions/41906206/pip-unicodedecodeerror-utf8-codec-cant-decode-byte
I'm unfortunately not sure what we can do about this, we're reliant on pip
for Python builds, and this doesn't seem to come from any pass-through behavior. Does it behave differently if you run pip
directly?
Thanks for coming back to me. You are right it was an encoding issue of the requirements.txt file. To check it I opened it and printed the encoding info.
f=open('requirements.txt, 'w') print(f.encoding)
The result was cp1252.
I altered this by changing it in the Windows control panel
Control panel (make sure you are set to category view)> Clock and Region > Region > Adminsitrative > Change System Locale
and ticked the box that says 'Beta: Use Unicode UTF-8 for worldwide language support'. Following a restart, my encoding had changed to 'cp65001' which I believe is equivalent to UTF8 and I was able to create my Lambda Function.
Glad you were able to resolve this! FWIW I've seen this periodically with Python builds especially, pip
seems to be very sensitive to encoding issues. Unfortunately, nothing we can do on our end about that.
When I sam build
I get the bug, but when I pip install -r requirements.txt
I don't get the bug.
Thus, it seems to me to be a SAM CLI issue.
OS: MacOS Catalina 10.15.7 python: 3.9.7 (installed via pyenv) pip: 21.2.3 (paired with python 3.9.7 installed via pyenv) sam cli: 1.33.0
My requirements.txt file:
atomicwrites==1.4.0; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0"
attrs==21.2.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
colorama==0.4.4; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.5.0"
docopt==0.6.2
iniconfig==1.1.1; python_version >= "3.6"
packaging==21.2; python_version >= "3.6"
pluggy==1.0.0; python_version >= "3.6"
py==1.11.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
pyparsing==2.4.7; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
pytest-watch==4.2.0
pytest==6.2.5; python_version >= "3.6"
python-dateutil==2.8.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.3.0")
python-util==1.2.1
six==1.16.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0"
toml==0.10.2; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
watchdog==2.1.6; python_version >= "3.6"
sam build
Building codeuri: /Users/REDACTED/py-root runtime: python3.9 metadata: {} architecture: x86_64 functions: ['VroHtnDataProcessor']
Running PythonPipBuilder:ResolveDependencies
Build Failed
Error: PythonPipBuilder:ResolveDependencies - {docopt==0.6.2(sdist), pytest-watch==4.2.0(sdist)}
mkdir pip-install-test/
cp requirements.txt pip-install-test/
cd pip-install-test/
pyenv local 3.9.7
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Produces output (which looks ok to me, but I'm not the most advanced python programmer)
Ignoring atomicwrites: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.5.0"' don't match your environment
Collecting attrs==21.2.0
Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting docopt==0.6.2
Using cached docopt-0.6.2.tar.gz (25 kB)
Collecting iniconfig==1.1.1
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging==21.2
Using cached packaging-21.2-py3-none-any.whl (40 kB)
Collecting pluggy==1.0.0
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting py==1.11.0
Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting pyparsing==2.4.7
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pytest-watch==4.2.0
Using cached pytest-watch-4.2.0.tar.gz (16 kB)
Collecting pytest==6.2.5
Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting python-dateutil==2.8.2
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting python-util==1.2.1
Using cached python_util-1.2.1-py3-none-any.whl (11 kB)
Collecting six==1.16.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting toml==0.10.2
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting watchdog==2.1.6
Using cached watchdog-2.1.6-cp39-cp39-macosx_10_9_x86_64.whl (85 kB)
Collecting colorama>=0.3.3
Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pytest-watch, since package 'wheel' is not installed.
Installing collected packages: pyparsing, toml, py, pluggy, packaging, iniconfig, attrs, watchdog, six, pytest, docopt, colorama, python-util, python-dateutil, pytest-watch
Running setup.py install for docopt ... done
Running setup.py install for pytest-watch ... done
Successfully installed attrs-21.2.0 colorama-0.4.4 docopt-0.6.2 iniconfig-1.1.1 packaging-21.2 pluggy-1.0.0 py-1.11.0 pyparsing-2.4.7 pytest-6.2.5 pytest-watch-4.2.0 python-dateutil-2.8.2 python-util-1.2.1 six-1.16.0 toml-0.10.2 watchdog-2.1.6
WARNING: You are using pip version 21.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/REDACTED/venv/bin/python3 -m pip install --upgrade pip' command.
I think I was able to get little more insight into the issue.
MacOS + pyenv
❯ head -n 1 `which sam`
#!/usr/local/Cellar/aws-sam-cli/1.37.0/libexec/bin/python3.8
so the expectation would be following to work
/usr/local/Cellar/aws-sam-cli/1.37.0/libexec/bin/pip3.8 install wheel
After some digging around did this:
edit /usr/local/Cellar/aws-sam-cli/1.37.0/site-packages/aws_lambda_builders/workflows/python_pip/packager.py
line 658
before or after invoke_pip = [self.python_exe, "-c", exec_string]
added print(self.python_exe)
to my surprise it was pointing to /Users/XYZ/.pyenv/shims/python3.8
ran /Users/XYZ/.pyenv/shims/pip3.8 install wheel
and then sam build
and it worked.
Please check and confirm here if this works for you too.
same issue for jsonschema.
Build Failed Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.18.1(wheel)}
Same issue with autobahn (although this can happen with any dependency):
Error: PythonPipBuilder:ResolveDependencies - {autobahn==22.3.2(sdist)}
In my case the used version was not compatible (22.3.2). I updated my requirements.txt with:
autobahn==21.2.1
and then run:
pip3 install -r requirements.txt
I found the working version after some searching in the repo of the main package using this dependency but this worked for me.
@
I solved this using
sam build --use-container
, however this works only when template.yml file is present. Using the graphical form does not allow to insert --use-container and it fails, so I do not count this as a solution, but workaround instead... // Also since I am unable to build app locally in my IDE, I am unable to run/debug it, so it makes no sense to use this workaround.
how do you test it locally after building with container?
Just add some extra insights here.
sam project structure as follows: |- template.yaml |- samconfig.toml HelloWorld --- |- requirements.txt --- |- mymodule ------- |- setup.py ------- |- setup.cfg ------- |- src ------------|- code --------------- |- __init__.py --------------- |- code.py
I am using a local module and declare it as "-e relative/path" at requirements.txt.
The problem is that the relative path declared is very tricky.
When sam build --use-container
from the project root, it should be -e mymodule/
. Otherwise it would cause PythonPipBuilder:ResolveDependencies
, yet -e mymodule/
wouldn't work the same when sam build
from the project root.
This fixed the ResolveDependencies error for me
pip install wheel
This definitely worked for me. But, could someone explain why is this needed? It would be great if it worked out of the box
My set of tricks to make it work on Apple M1 (MacOS 11.6 Big Sur)
brew remove aws-sam-cli # will be installed below using pip
brew install pyenv
# Also you may need to remove an existing python version.
# I had to remove and reinstall it back later after all other commands to make things work
# brew remove python@3.9
pyenv install 3.7.13 # < 3.7 is currently required by AWS SAM
pyenv global 3.7.13
vim ~/.zshrc || vim ~/.bashrc
### add these lines to the *rc file:
# >> PYENV
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"
# << PYENV
#save and close the file
#open a new terminal window
pip install aws-sam-cli wheel
# you may also need this if you have Podman, even if you build locally:
# pip install paramiko
Check versions:
python -V && pip -V && sam --version
# Python 3.7.13
# pip 22.0.4 from /Users/yevhen.hlotov/.pyenv/versions/3.7.13/lib/python3.7/site-packages/pip (python 3.7)
# SAM CLI, version 1.53.0
Build the project with sam build
and it should work now.
Although, it still cannot build in containers when using Podman due to the separate issue.
To anybody who sees this in the future, the I was having the same issue as this thread (and like others, I was able to resolve using sam build --use-container
. It turns out that I didn't have a virtual environment set up in my project root, which makes sense why this would create dependency issues.
I only figured this out because my deployment package became too large to deploy, and I realized it was bundling a ton of packages I didn't need. This led me to realize I didn't have an virtual env configured.
I am still seeing this issue with sam build --use-container. Using podman on macOS (tried PKG, homebrew, and now pyenv. Using DOCKER_HOST. sam local invoke and sam local start-api only work with ---skip-pull-image. sam build without use-container works
Build Failed Running PythonPipBuilder:ResolveDependencies Error: PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: requests
trying the hello world example
On the surface, my problem seemed to be the same as many others here. But it turns out that my issue was an expired token to our CodeArtifact repository (which we use for all our Python dependencies). This became evident when using the --debug
option and seeing that sam build
was unable to download any dependencies.
I have the same
Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - {pendulum==2.0.5(sdist)}
make: *** [build] Error 1
I have the same, but for another package: lightgbm 4.2.0
Final incompatible: {lightgbm==4.1.0(wheel)} Final missing wheels: {lightgbm==4.1.0(sdist)} PythonPipBuilder:ResolveDependencies failed
I had to downgrade my lgbm version to 3.3.5 and it the build worked, testing locally too.
Description
Trying to create Python Lambda function from PyCharm not working
Steps to reproduce
Invoke Lambda creation using "Create new AWS Lambda..." Runtime: Python 3.7 requirements.txt: https://pastebin.com/HpU1it63 Run using strict template file: template.yml:
Observed result
Used graphical create, not CLI, error:
Full log: https://pastebin.com/asgCa9D4
Expected result
Successful build
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
OS: macOS Mojave 10.14.6 SAM: SAM CLI, version 0.43.0 P: Python 3.7.6 IDE: PyCharm 2019.3 AWS: aws-cli/2.0.1 Python/3.7.4 Darwin/18.7.0 botocore/2.0.0dev5 used wheel: wheel 0.34.2 used pip: pip 19.0.3