Closed mebibou closed 3 years ago
Hi @mebibou, This error means that your own package is too big (when unzipped). From my understanding, the geolambda layer must weight around 120Mb (unzipped) so you need to make sure your package weight less than 130Mb (unzipped) (130Mb + 120Mb == 250Mb which is the limit)
Ok that makes sense, I guess its because I am using some libraries that import numpy
on their own as well and not sure how to make sure I don't double import it. That aside, I managed to go around the limitation by using zip: true
in the serverless.yml
(which zipped the dependencies), but now I am getting this error when running the function:
[Errno 2] No such file or directory: '/usr/bin/ogr2ogr': '/usr/bin/ogr2ogr'
I did set GDAL_DATA: /opt/share/gdal
in my env though.
My full serverless.yml
:
service: helloworld
provider:
name: aws
runtime: python3.7
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: non-linux
zip: true
package:
exclude:
- '**/*'
include:
- '*.py'
functions:
hello:
handler: handler.handle
events:
- http:
path: /
method: post
layers:
- arn:aws:lambda:us-east-1:552188055668:layer:geolambda-python:3
environment:
GDAL_DATA: /opt/share/gdal
PROJ_LIB: /opt/share/proj
Oh so I actually found how to change the path where to find the ogr2ogr
bin, because I am using pygdaltools
I had to set this at the beginning of the handler file:
import gdaltools
gdaltools.Wrapper.BASEPATH = '<path>'
but I've tried with /opt/share
and /opt/share/gdal
but I still get the error No such file or directory: '/opt/share/ogr2ogr'
or No such file or directory: '/opt/share/gdal/ogr2ogr'
Where is the ogr2ogr
file in the layer exactly?
Hello @mebibou sorry for the delay.
ogr2ogr
is not in the Layer, none of the GDAL binaries are included, just the libraries.
So all you need to do is copy ogr2ogr
from the GeoLambda docker image (https://hub.docker.com/repository/docker/developmentseed/geolambda) into your own deployment package, in which case it will be in under /var/task
Ok so what I did:
docker pull developmentseed/geolambda:latest
docker run --rm --entrypoint /bin/sh developmentseed/geolambda:latest -c "cat /usr/local/bin/ogr2ogr" > ogr2ogr
chmod 755 ogr2ogr
Then in my serverless.yml
I included the library ogr2ogr
(in the package/include
part), modified the code to found the binary in my python file to gdaltools.Wrapper.BASEPATH = '/var/task'
.
Now running the function gives a different error:
(127, b'/var/task/ogr2ogr: error while loading shared libraries: libgdal.so: cannot open shared object file: No such file or directory\\n')
Do I need to also pull all the libraries missing? that doesnt sound right
Hi @mebibou sorry this got dropped. Did you figure this out? You should be installing ogr2ogr from within the GeoLambda container, rather than using the one from your local machine which may be using a GDAL compiled differently than the one in the container.
When using the layer
I cannot deploy to Lambda because I will get this error:
How can I use the layer on python 3 without having a limit problem?