CaseGuide
commented
4 years ago I note that the reasoning for using a tarball is discussed in https://github.com/Miserlou/Zappa/pull/1022 and https://github.com/Miserlou/Zappa/pull/1037. Since the file slash issue is fundamental to the tarfile python module, and there are good reasons for using a tarfile (streamed unzipping to/tmp seems to be the primary one), I'll go down the "fix the slashes in the tarball" path.
I confirmed that the line tarinfo = tarfile.TarInfo(os.path.join(root.replace(temp_project_path, '').lstrip(os.sep), filename)) in core.py is a source of this. It looks like replacing the separators in the string and path properties of the TarInfo object might fix it.
Livestream of edits as I try stuff...
- I downloaded the current (0.51.0) Zappa code zip
pip uninstall zappa- nav to zappa code,
pip install -e . - Got invalid characters in Readme, erased readme,
pip install -e .success.
Replace the slashes in the tarball
Tried replacing the slashes in zappa\cli.py:
with open(os.path.join(root, filename), 'rb') as f:
if os.sep == '\\':
tarinfo.path = tarinfo.path.replace(os.sep, '/')
archivef.addfile(tarinfo, f)Listing the tar contents in WSL now shows the correct slashes:
-rwxr-xr-x 0/0 342624 2020-07-29 07:38 pandas/_libs/tslibs/tzconversion.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x 0/0 388 2020-07-29 07:38 pandas/_libs/tslibs/__init__.py
-rwxr-xr-x 0/0 69 2020-07-29 07:38 pandas-0.25.1.dist-info/entry_points.txt
-rwxr-xr-x 0/0 4 2020-07-29 07:37 pandas-0.25.1.dist-info/INSTALLER
-rwxr-xr-x 0/0 1582 2020-07-29 07:37 pandas-0.25.1.dist-info/LICENSEHowever I am getting the same error.
Turn it off and then on again
Trying zappa undeploy production --remove-logs then zappa deploy because turning it on and off again definitely applies to software development.
Still failed with [1596025336429] Unable to import module 'index': No module named 'index' in tail.
Make it a zip file instead of a tar, performance decrease be damned.
Edited core.py to use a zip, changed the file extensions in core.py filenames form .tar.gz to .zip.
Edited handler.py to use zipfile if .zip in name.
It looks like many errors were begin suppressed when I had "lambda_handler": "index.handler",. Removing that revealed some issues with unzipping the file.
Hooray a new error, now I'm running out of disk space! OSError: [Errno 28] No space left on device
Forked my repo and made a lighter weight venv
The repo is now right on the edge of size for needing slim_handler. I successfully deployed the code with "slim_handler": false, but am confident that the next few bits of work I will have to do will cross the threshold.
Still unable to deploy with slim_handler. When I deploy with slim_handler true and lambda_handler commented out I see an error that appears to be related to this issue https://github.com/Miserlou/Zappa/issues/1834#issuecomment-632994410
Adding "include" : [] didn't fix the issue. I attempted the fix described in #1834 adding the following to /zappa/init.py:
import pymysql
pymysql.install_as_MySQLdb()
pymysql.version_info = (1, 3, 13, 'final', 0)....even though this project doesn't use a DB. Was greated with a new error:
Unable to import module 'handler': attempted relative import with no known parent package
LaundroMat
It is not possible to use Zappa on Windows with slim_handler = true. This is likely due to the slashes used when tarring the file. This issue is well known, probably easy to fix, and has cropped up in #1870 and #1358 with a PR for #1358 in #1570 that seemed to not get integrated.
Context
Using venv and Python 3.6 on Widows 10, I was able to successfully deploy a project and get the expected results. Upon installing pandas, my lambda project exceeded what Lambda would accept. After
pip install pandasand the failed deployment I changes zappa_settings.json to addslim_handler : true, then deployed. I'm now encountering an issue returning the errorerrorMessage": "Unable to import module 'index'". Myhandlerfunction is a regular python function that is inindex.pyand was otherwise working fine.Expected Behavior
Expect this deployment to be successful and return a 200 and some JSON via my API Gateway
Actual Behavior
I receive a 502 {"message": "Internal server error"} from the API Gateway endpoint and
zappa tailreturns many instances of:[1596019020990] Unable to import module 'index': No module named 'index'. My function is in index.py."lambda_handler": "index.handler",Possible Fix
Steps to Reproduce
"slim_handler": true, "delete_local_zip": false,zappa tailexpected output should be something like:Unable to import module <handler_function>: No module named <your import>tar --list --verbose --file=<project>-<env>-<time>.tar.gzon Linux. The output, in my case from WSL Ubuntu, proves that there are indeed mixed slashesYour Environment
pip freeze: I have a bloated pip freeze here, I'm not actually using django for this particular function, but its part of a set of lambdas that are all on the same venv. Posted my requirements.txt for brevity:Django==2.0.6 mysqlclient==1.3.12 pymysql
Created by djangocms-installer
django-cms>=3.7,<3.8 djangocms-admin-style>=1.5,<1.6 django-treebeard>=4.0,<5.0 djangocms-text-ckeditor>=3.7,<4.0 djangocms-link>=2.5,<2.7 djangocms-icon>=1.4,<1.6 djangocms-style>=2.2,<2.4 djangocms-googlemap>=1.3,<1.5 djangocms-snippet>=2.2,<2.4 djangocms-video>=2.1,<2.4 djangocms-file>=2.3,<2.5 djangocms-picture>=2.3,<2.5 djangocms-bootstrap4>=1.5,<1.7 easy_thumbnails django-filer>=1.3 django-classy-tags>=0.9 django-sekizai>=1.0 django-mptt>0.9 html5lib>=1.0.1 Pillow>=3.0 six pytz
Added later
djangorestframework django-s3-storage==0.13.2
What I'm using for this part of the project.
selenium==3.13.0 pandas==0.25.1 # This version of pandas is needed for compatibility with zappa's python-dateutils requirement python-dateutil==2.6.1 # This overlaps the reqs of zappa 0.51.0 and pandas 0.25.1
{ "dev": { "aws_region": "us-east-1", "profile_name": "default", "project_name": "",
"runtime": "python3.6",
"s3_bucket": "",
"use_precompiled_packages": true,
"lambda_handler": "index.handler",
"memory_size": 2048,
"timeout_seconds": 300,
"cloudwatch_log_level": "INFO",
"slim_handler": true,
"delete_local_zip": false, // Delete the local zip archive after code updates. Default true.
"delete_s3_zip": false // Delete the s3 zip archive. Default true.
}
}