search

serverless / serverless

⚡ Serverless Framework – Effortlessly build apps that auto-scale, incur zero costs when idle, and require minimal maintenance using AWS Lambda and other managed cloud services.
https://serverless.com
MIT License
46.44k stars 5.72k forks source link

invoke local not finding python module dateutil #5115

Closed jacobevans closed 5 years ago

jacobevans commented 6 years ago

This is a Bug Report

Description

I have a aws/python3.6 serverless project (with serverless-python-requirements). I've added other libraries with no issue, but I'm trying to use the python-dateutil module, and experiencing issues with sls invoke local, and only sls invoke local.

When I try to invoke a function locally, I get this error:

$ sls invoke local -f function_name
Traceback (most recent call last):
  File "/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/invoke.py", line 60, in <module>

    module = import_module(args.handler_path.replace('/', '.'))
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./handler.py", line 2, in <module>
    import dateutil.parser
ModuleNotFoundError: No module named 'dateutil'

This is the relevant line at the beginning of my handler.py:

import dateutil.parser

However, if I deploy and do a regular sls invoke i get no such error. Additionally if I open a python shell and try the exact same import statement, it works just fine.

I'm on a mac, using homebrew, and a venv created with: python3 -m venv

versions, etc:

$ which python3
/Users/Jacob/dev/gal-autoembed/venv/bin/python3

$ python3 --version
Python 3.6.4

$ sls --version
1.28.0

$ cat package.json
{
  "name": "gallery-autoembed",
  "description": "",
  "version": "0.1.0",
  "dependencies": {},
  "devDependencies": {
    "serverless-python-requirements": "^4.1.0"
  }
}

$ cat requirements.txt
certifi==2018.4.16
chardet==3.0.4
docopt==0.6.2
idna==2.7
jedi==0.12.0
parso==0.2.1
prompt-toolkit==1.0.15
ptpython==0.41
Pygments==2.2.0
python-dateutil==2.7.3
requests==2.19.1
six==1.11.0
urllib3==1.23
wcwidth==0.1.7

$ ls -l venv/lib/python3.6/site-packages/
total 112
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 Pygments-2.2.0.dist-info
drwxr-xr-x   5 Jacob  staff    170 Jul 10 17:18 __pycache__
drwxr-xr-x   7 Jacob  staff    238 Jul 10 17:18 certifi
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 certifi-2018.4.16.dist-info
drwxr-xr-x  43 Jacob  staff   1462 Jul 10 17:18 chardet
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 chardet-3.0.4.dist-info
drwxr-xr-x  14 Jacob  staff    476 Jul 10 17:18 dateutil
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 docopt-0.6.2.dist-info
-rw-r--r--   1 Jacob  staff  19946 Jul 10 17:18 docopt.py
-rw-r--r--   1 Jacob  staff    126 Jul 10 17:16 easy_install.py
drwxr-xr-x  11 Jacob  staff    374 Jul 10 17:18 idna
drwxr-xr-x   8 Jacob  staff    272 Jul 10 17:18 idna-2.7.dist-info
drwxr-xr-x  15 Jacob  staff    510 Jul 10 17:18 jedi
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 jedi-0.12.0.dist-info
drwxr-xr-x  13 Jacob  staff    442 Jul 10 17:18 parso
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 parso-0.2.1.dist-info
drwxr-xr-x   7 Jacob  staff    238 Jul 10 17:18 pip
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 pip-10.0.1.dist-info
drwxr-xr-x   6 Jacob  staff    204 Jul 10 17:16 pkg_resources
drwxr-xr-x  35 Jacob  staff   1190 Jul 10 17:18 prompt_toolkit
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 prompt_toolkit-1.0.15.dist-info
drwxr-xr-x  20 Jacob  staff    680 Jul 10 17:18 ptpython
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 ptpython-0.41.dist-info
drwxr-xr-x  22 Jacob  staff    748 Jul 10 17:18 pygments
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 python_dateutil-2.7.3.dist-info
drwxr-xr-x  21 Jacob  staff    714 Jul 10 17:18 requests
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 requests-2.19.1.dist-info
drwxr-xr-x  34 Jacob  staff   1156 Jul 10 17:16 setuptools
drwxr-xr-x  12 Jacob  staff    408 Jul 10 17:16 setuptools-28.8.0.dist-info
drwxr-xr-x   9 Jacob  staff    306 Jul 10 17:18 six-1.11.0.dist-info
-rw-r--r--   1 Jacob  staff  30888 Jul 10 17:18 six.py
drwxr-xr-x  16 Jacob  staff    544 Jul 10 17:18 urllib3
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 urllib3-1.23.dist-info
drwxr-xr-x   8 Jacob  staff    272 Jul 10 17:18 wcwidth
drwxr-xr-x  10 Jacob  staff    340 Jul 10 17:18 wcwidth-0.1.7.dist-info

Additional Data

dschep commented 6 years ago

Is the virtualenv active? serverless-python-requirements is only for deployment, and sls invoke local only uses a virtualenv if it's active by using the VIRTUAL_ENV environment variable. It can't magically find it, since it could be anywhere.

jacobevans commented 6 years ago

@dschep Yes, sorry, I should have specified. it is activated with source venv/bin/activate (venv is where the venv is). I've double and triple checked that.

Also, I don't believe I'd be able to import the library in the python shell otherwise.

$ which python
.../projectdir/venv/bin/python
jacobevans commented 6 years ago

So, it does seem like the invoke local is somehow not operating inside of the virtualenv properly. if I add the following to the top of handler.py, it makes it work:

import sys
sys.path.append('./venv/lib/python3.6/site-packages')

Here is what sys.path looks like before I append that path:

['/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal', 
'/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', 
'/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6', 
'/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
'.']

For comparison, here's what I see if I print sys.path in a python shell that I open inside my project directory with the venv activated:

['', 
'/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python36.zip',
'/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6', 
'/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', 
'/full/path/to/projectdir/venv/lib/python3.6/site-packages']
dschep commented 6 years ago

I've figured out the issue. The venv module doens't create a binary at $VIRTUAL_ENV/bin/pythonX.Y like virtualenv does. I'm creating PR to fix it :smile:

dschep commented 6 years ago

As a work around for now, create a python3.6 symlink:

cd venv/bin
ln -s python3 python3.6
ademidun commented 4 years ago

Not sure if this is related to the problem mentioned above but If anyone comes here through Google try this:

npm install serverless --save-dev; ./node_modules/serverless/bin/serverless invoke local --function transferContent --data '{"content_type":"scholarship"}'