rackerlabs / lambda-uploader

Helps package and upload Python lambda functions to AWS
Apache License 2.0
270 stars 56 forks source link

Requirements not being copied to zip #129

Closed wobeng closed 7 years ago

wobeng commented 7 years ago

When I specify pip libraries in the config or requirements.txt It looks like it installs it but does not copy to the zip file

Is there something I'm doing wrong? I have attached the zip, where there is no lib files

lambda_function.zip


λ Building Package
INFO:lambda_uploader.package:Building new virtualenv and installing requirements
DEBUG:lambda_uploader.package:Virtualenv stdout: Using real prefix '/usr'
New python executable in /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/venv/bin/python2
Also creating executable in /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/venv/bin/python
Installing setuptools, pip, wheel...done.
Running virtualenv with interpreter /home/testuser/py/env2/bin/python2

DEBUG:lambda_uploader.package:Virtualenv stderr:
DEBUG:lambda_uploader.package:Installing requirements found [u'boto3', u'requests', u'validators', u'Jinja2'] in config
DEBUG:lambda_uploader.package:Pip stdout: Requirement already satisfied: boto3 in /home/testuser/py/env2/lib/python2.7/dist-packages
Requirement already satisfied: requests in /home/testuser/py/env2/lib/python2.7/dist-packages
Requirement already satisfied: validators in /home/testuser/py/env2/lib/python2.7/dist-packages
Requirement already satisfied: Jinja2 in /home/testuser/py/env2/lib/python2.7/dist-packages
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/testuser/py/env2/lib/python2.7/dist-packages (from boto3)
Requirement already satisfied: botocore<1.5.0,>=1.4.1 in /home/testuser/py/env2/lib/python2.7/dist-packages (from boto3)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.10 in /home/testuser/py/env2/lib/python2.7/dist-packages (from boto3)
Requirement already satisfied: decorator>=3.4.0 in /home/testuser/py/env2/lib/python2.7/dist-packages (from validators)
Requirement already satisfied: six>=1.4.0 in /home/testuser/py/env2/lib/python2.7/dist-packages (from validators)
Requirement already satisfied: MarkupSafe>=0.23 in /home/testuser/py/env2/lib/python2.7/dist-packages (from Jinja2)
Requirement already satisfied: docutils>=0.10 in /home/testuser/py/env2/lib/python2.7/dist-packages (from botocore<1.5.0,>=1.4.1->boto3)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/testuser/py/env2/lib/python2.7/dist-packages (from botocore<1.5.0,>=1.4.1->boto3)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /home/testuser/py/env2/lib/python2.7/dist-packages (from s3transfer<0.2.0,>=0.1.10->boto3)

DEBUG:lambda_uploader.package:Pip stderr:
INFO:lambda_uploader.package:Copying site packages
INFO:lambda_uploader.utils:Copying source files
INFO:lambda_uploader.package:Copying lib64 site packages
INFO:lambda_uploader.utils:Copying source files
INFO:lambda_uploader.utils:Copying source files
DEBUG:lambda_uploader.utils:Copying /home/testuser/py/ca-lambda/test/lambda.json to /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/lambda_package/
DEBUG:lambda_uploader.utils:Copying /home/testuser/py/ca-lambda/test/hello.py to /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/lambda_package/
DEBUG:lambda_uploader.utils:Copying /home/testuser/py/ca-lambda/test/requirements.txt to /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/lambda_package/
INFO:lambda_uploader.package:Creating zipfile
DEBUG:lambda_uploader.package:Zipping /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/lambda_package/lambda.json as lambda.json
DEBUG:lambda_uploader.package:Zipping /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/lambda_package/hello.py as hello.py
DEBUG:lambda_uploader.package:Zipping /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/lambda_package/requirements.txt as requirements.txt```
martinb3 commented 7 years ago

Hello! It seems like you have a lot of packages that pip is refusing to install because they are installed globally in your Python (see the part about Requirement already satisfied). You probably want to ship your entire set of installed Python libraries as a Lambda function, so I'd recommend uninstalling them globally so pip can install them locally for the project and virtualenv can capture them.

wobeng commented 7 years ago

Hi martin,

Thanks for responding. env2 is a virtualenv not global

martinb3 commented 7 years ago

Hi @wobeng -- /home/testuser/py/ca-lambda/test/.lambda_uploader_temp/venv is a brand new virtualenv, so it shouldn't be able to see env2 unless you're actively loading it in all new shells/somehow making it globally activated. Is it possible you've got it loading automatically?

Lambda-uploader doesn't do anything beyond running virtualenv to create a separate environment, so for some reason, additional virtual environments aren't letting you install your dependencies. You could troubleshoot this yourself by attempting the same...

wobeng commented 7 years ago

I had to rebuilt my linux environment, worked after that