Miserlou / Zappa

Serverless Python
https://blog.zappa.io/
MIT License
11.89k stars 1.2k forks source link

Do not upload a macOS .so file when a manylinux wheel is not available #914

Open marcoesposito1988 opened 7 years ago

marcoesposito1988 commented 7 years ago

Context

I tried to upload an app using pycryptodome from macOS, and i got a cryptic `OSError: Cannot load native module 'Crypto.Cipher._raw_ecb'` error. Doing the same from Kubuntu worked fine. I inspected the zip created by `zappa package`, and I noticed that a macOS .so file (`_raw_ecb.cpython-36m-darwin.so`) ended up in the tarball. Actually, all of them: the whole macOS wheel was copied into it. I guess that this was the result of the fallback action, since pycryptodome is not in lambda-packages and does not have a manylinux wheel in PyPI. I will try to compile it and submit it for lambda-packages, but having a sanity check before copying a wheel would not be bad: it should not be done when there is a .so file and the OS is mac. Actually, having #542 merged would probably help a lot with these kind of errors. ## Expected Behavior
  1. Create an app using pycryptodome (or any other library using native DLLs)
  2. Deploy it from macOS
  3. The lambda should work

Actual Behavior

  1. Create an app using pycryptodome (or any other library using native DLLs)
  2. Deploy it from macOS
  3. OSError: Cannot load native module 'INSERT_YOUR_DLL_NAME_HERE' because a macOS DLL gets uploaded to lambda (which is linux)

Possible Fix

Steps to Reproduce

  1. Create a virtualenv containing Flask-User and an app importing it
  2. zappa {init,deploy} under macOS
  3. zappa tail
Miserlou commented 7 years ago

This problem is not as easy as it sounds, as we'll have to actually inspect every file to see if it should be included or not, as some people may have manually created so files they want in their package. I think there are libraries and unix utilities to do this, but I don't think they work on Windows, for example.

marcoesposito1988 commented 7 years ago

Maybe one solution could be only allowing to use an explicit requirements.txt as in #542 when not on linux.

And allow to force it in case an advanced user copies the .so file from a linux box by hand.

exp0nge commented 6 years ago

@marcoesposito1988 What did you end up doing?

bjinwright commented 6 years ago

Try this https://github.com/capless/python-jose-cryptodome/blob/master/README.rst

gallamine commented 6 years ago

@bjinwright I tried using your python-jose-cryptodome in a zappa/lambda install (actually I'm using warrant) but I keep getting similar errors to the OP:

OSError: Cannot load native module 'Crypto.Hash._SHA256'