Open mbox opened 7 years ago
I think we'd need to do something like this in the handler: https://docs.python.org/2.7/library/ctypes.html
The problem is knowing what to load.
I think the general consensus after discussion in Slack is that certain file-types (.so) should be left in the main package so that they're in the right place for the rest.
Are there any known workarounds for this at the moment?
So it looks like the include
setting can be used to solve this, by specifying a path relative to the handler's base directory. If you have libraries in lib/
you would use "include": ["lib/mylibrary.so"]
.
Note that order of imports is very important here. When loading a library X that depends on a library Y, make sure to load Y first. The current version of zappa provides very little feedback on failure -- "Failed to find library...right filename?"
is shown in all cases and the actual exception is hidden.
Also this won't help if any of your python modules use ctypes.utils.find_library
. In that case your best bet is to also set LD_LIBRARY_PATH
and use a python3.6 lambda
I think this issue might have been solved by #789
This error isn't very helpful either "Failed to find library...right filename?" I'd rather get the original error or atleast a library or filename.
I'm experiencing this issue trying to deploy a lambda function that uses pandas. I can't deploy without slim_handler because the unzipped size is too large, but pandas doesn't work when slim_handler is true.
Similar issue, I think python Babel put my deployment over the limit. Just getting timeouts and "Failed to find library...right filename?". Any advice on how to proceed or even debug?
@zgazak @cscanlin Have you found a solution to this problem or method for debugging since your comment here?
I am having this same issue. Need to use slim_handler due to file size but shared libraries (.so files) are not being found.
It seems I'm having the same problem while deploying chromedriver and headless-chromium, they just crash and have this error "Failed to find library...right filename?" before the crash. Has anyone figured out how to fix this?
I am also having this issue.
Surely this is an incredible oversight in the design of slim_handler?
Is there any way around this at all?
I wasn't able to solve this, @themmes
Where do we go from here?
This is clearly a long-standing issue and essentially removes all the usefulness of slim_handler.
@GeoTob Have you had any more success with this?
@ckyleda it's been too long since I last looked at it.
From memory using the include
setting worked in most cases (see my earlier comment).
I think we also patched the library to return more useful debugging output than Failed to find library...right filename?
. That code is here https://github.com/Miserlou/Zappa/blob/4cf80b2a0212a5a8111f0f66348b46ef39507ed1/zappa/handler.py#L117 and you can quite easily overwrite it locally to tell you exactly what library it's looking for.
print ("Failed to find library {} ...right filename?").format(library)
I set mine to the above locally. Thanks for the tip!
Any workaround on this?
@GeoTob Bit new to zappa - question about the include setting.
I'm using slim_handler
and getting libpq.so.5: cannot open shared object file: No such file or directory: ImportError
. I believe libpq.so.5 is a shared library installed with postgres client.
If my virtualenv is at ~/.virtualenvs/
Where do I need to put the libpq.so.5? What path do I put in the include setting?
Hello guys,
My apollogies if I am overlooking something obvious but I had a similar issue with shared libraries
and slim_handler
and I would like to share the solution that worked for me - maybe it 'll help you.
AWS Lambda service offers a feature called Layers. Basically it seems to be an option to use when large binary files (like shared libs) need to be available at runtime in the lambda function. A fork of Zappa by @jedestep supports "attaching" layers to your lambda setting a related option in your zappa_settings.json
.
I am using Geodjano which requires quite a few geospatial libraries (.so files) to be available to django and uploading them in a layer which made them available through the /opt/lib/
path solved my problem. If you are interested in more info check the related issue #985.
I did not make much testing with slim_handler
set to true but a deployment I made few minutes ago seemed to find all libraries. Again sorry if I am missing something and this approach is not suitable for this issue.
I ran into a related issue setting up python3-saml on with the dependent python-xmlsec
import failing: ImportError: libxmlsec1.so.: cannot open shared object file: No such file or directory
. (I also had slim_handler
set to true.)
In order to get lambda to load the dependencies I first set the LD_LIBRARY_PATH
to lib:$LD_LIBRARY_PATH
as suggested by this set of tips (which as a newbie to lambda I found helpful). Then set the include
parameter like @GeoTob suggests in your zappa_settings file to path of the dependency (e.g. "include": ["/var/task/lib"]
), and drop the dependencies in a lib
folder at the root from which you are calling zappa.
I just wanted to add something that helped me solve my issues with binary libraries not getting referenced or missing during packaging.
apt-get update && apt-get install -y build-essential <insert required libs here> && ldconfig -v
STATIC_DEPS=true
changes how your pip install links to the binary libraries. Every package builds and retains the required binary libraries locally. This is ideal for zappa packaging and lambda deployment. For example: STATIC_DEPS=true pip install -r requirements.txt
lxml
, stop wasting your time trying to figure out how to install it. The problem is with zappa's lambda-packages portion of the packaging process. It overwrites some libraries that thinks it has a better version of. In this case, it tries to install a broken lxml
package. I wasted days before figuring this out. Solution: inside zappa_settings.json, use use_precompiled_packages=false
I hope this helps someone out there. My problem stemmed out of python3-saml
where it kept breaking looking for missing *.so files, and then eventually got stuck at a missing lxml.etree
module. Also pay attention to the Python runtime version. 3.6, 3.7, and 3.8 all had varying errors that ended up at that lxml
issue.
Context
Normally, anything in the /lib directory will end up in `/var/task/lib` which Lambda includes on `LD_LIBRARY_PATH` which lets code link to it. However, when `slim_handler` is in effect, the code lands up at `/tmp/Either there should be a way to say "when building the slim_handler, include these libraries in
/lib
", or there should be a symlink in the slim_handler package to the "real" location in the project files on/tmp
Actual Behavior
You can't get binary libraries to work
Possible Fix
Steps to Reproduce
lib
subdirectorylib
slim_handler
totrue
lib
slim_handler
to falseYour Environment
zappa_settings.py
: