libvips / pyvips

python binding for libvips using cffi
MIT License
646 stars 49 forks source link

import pyvips throws exception for python 2.7 and 3.6 in AWS lambda (ZLIB_1.2.9 not found while loading libvips.so.42) #42

Open munirhossain opened 6 years ago

munirhossain commented 6 years ago

I'm writing a AWS Lambda function using pyvips python package. But it throws exception when import pyvips is executed. The error is as follows. I've zipped cffi, pycparser, pyvips and a lib directory which contains all the shared libraries of libvips for lambda build package. I've also set the environment variable LD_LIBRARY_PATH=./lib:LD_LIBRARY_PATH .The lambda.zip that I'm uploading to the AWS lambda can be found here.

cannot load library 'libvips.so.42': /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by ./lib/libpng16.so.16).  Additionally, ctypes.util.find_library() did not manage to locate a library called 'libvips.so.42': OSError
Traceback (most recent call last):
  File "/var/task/lambda.py", line 5, in lambda_handler
    import pyvips
  File "/var/task/pyvips/__init__.py", line 54, in <module>
    vips_lib = ffi.dlopen(_vips_libname)
  File "/var/task/cffi/api.py", line 141, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/var/task/cffi/api.py", line 802, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/var/task/cffi/api.py", line 797, in _load_backend_lib
    raise OSError(msg)
OSError: cannot load library 'libvips.so.42': /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by ./lib/libpng16.so.16).  Additionally, ctypes.util.find_library() did not manage to locate a library called 'libvips.so.42'

END RequestId: af867599-5876-11e8-a3ca-ffec3ebba19d
REPORT RequestId: af867599-5876-11e8-a3ca-ffec3ebba19d  Duration: 459.45 ms Billed Duration: 500 ms     Memory Size: 128 MB Max Memory Used: 21 MB  

This may not be a problem of libvips or pyvips but some setting in AWS Lambda function that I don't know. Created an issue with AWS as well.

jcupitt commented 6 years ago

Hello, I'll have a go at a docker build on aws.

jcupitt commented 6 years ago

This seems to work:

https://github.com/jcupitt/docker-builds/tree/master/pyvips-aws

Though I don't know if that'll help you much.

munirhossain commented 6 years ago

I think docker would be fine since it manages all its libraries. In my case I'm running a Lambda function without docker. And the problem is that even though I provide all the necessary libraries in lib directory and set the environment path LD_LIBRARY_PATH=./lib. It seems Lambda in unable to load all the libraries from this lib directory. I've found a work around which is running another process with my correct configuration set for the library path and load pyvips in that process only. It seems to work. It is a work around and the down side is that you need to create another process and manage it. So it comes with all the regular downside of it. But for now it is working fine. I mainly posted it here for the community and the AWS team to look into. This is not a VIPS problem and I'll update here if I find anything better.

baxhul commented 6 years ago

@munirhossain I am currently having the same problem. Did you manage to make it work? If not can you please explain your workaround?

baxhul commented 6 years ago

@munirhossain I managed to made it work. Had some fun replacing some dependencies from your package and at some point it worked:

https://drive.google.com/file/d/1dIuyObvLWlTFGDhzFS7EY5ikNNvNfHw1/view

munirhossain commented 6 years ago

Hi, Sorry for late reply. Good to know that you got it worked. If you don't mind please put a note in the git or somewhere how you fixed it. Please send me the link as well.

Best regards, Munir

On Mon, Oct 22, 2018, 1:55 AM baxhul notifications@github.com wrote:

@munirhossain https://github.com/munirhossain I managed to made it work. Had some fun replacing some dependencies from your package and at some point it worked:

https://drive.google.com/file/d/1dIuyObvLWlTFGDhzFS7EY5ikNNvNfHw1/view

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/libvips/pyvips/issues/42#issuecomment-431772063, or mute the thread https://github.com/notifications/unsubscribe-auth/AAzrIqtHyQPEs_qJxRdRDOCFn0OYfvktks5unYf1gaJpZM4UAK3q .

AntonOellerer commented 4 years ago

Hey @baxhul could you maybe report how you solved the issue in more detail? I am facing the same right now.