libgdal.so: cannot open shared object file: No such file or directory

[AnneSophie12]

Hello,

First of all, thank you for this awesome code that simplify this process so much !

Although, i'm pretty new with AWS Lambda and I've had trouble using it. I followed the python readme in order to build the docker (as i want to use it on eu-west-3).

For the requirements, i just need GDAL (for the use of gdalinfo and gdal_translate) and Pyproj so I removed rasterio and shapely from the example requirements.

I built the docker, run the package.sh and the python-package.sh and publish the 'deploy-layer.zip' to aws lambda.

I copied-paste the lambda handler from this repo into the lambda_funtion on the AWS Lambda Gui and linked my new layer.

But when i try to run the lambda function this message appears : "errorMessage": "Unable to import module 'lambda_function': libgdal.so: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError"

I tried different configuration of the requirements, dockerfiles, package.sh and python-package.sh in order to correct this but without really understanding what causes this error ..

Would you have an idea on how to help me ?

Thank you very much !!

[saheelBreezo]

Hey @AnneSophie12, make sure to set GDAL_DATA=/opt/share/gdal and PROJ_LIB=/opt/share/proj as environment variables on lambda. Also there will be two zips one in the root directory and other in the python directory. Both needs to be published as layers.

[matthewhanson]

@AnneSophie12 Sorry I've not been attending to this project for a while, but @saheelBreezo is correct, those fixes should take care of that error message.

Did you ever get it resolved?

[kwinwithak]

I'm getting the same error message as @AnneSophie12 after switching to v2.1.0-python:

[ERROR] Runtime.ImportModuleError: Unable to import module 'my_function_handler': libgdal.so: cannot open shared object file: No such file or directory

Prior to switching to the new v2.1.0-python layer, I was successfully using v2.0.0 without any errors like the one shown above. I was already setting GDAL_DATA=/opt/share/gdal and PROJ_LIB=/opt/share/proj as environment variables as suggested, so I don't think that's the problem. And because v2.1.0-python is supposed to contain everything in the v2.1.0 base layer, I'm only bringing in the python lambda layer to my function. Are there any additional instructions for using v2.1.0-python besides what's already provided in the readme?

[matthewhanson]

Hi @kwinwithak There actually isn't any deployed 2.1.0 Lambda layers yet....I built the image which is available on Docker Hub, but not yet pushed public Lambda layers. When I do that I'll update the ARNs in the README and create an official release.

[ferily7]

@matthewhanson I'm getting the same error as well but I am using the ARN for a lambda layer and adding it to my lambda function, plus my lambda function I'm using to read GDAL is in nodejs not Python.

I tried to set GDAL_DATA and PROJ_LIB as environmental variables on my lambda function (I just went to Environment Variables on the UI of the lambda function and added the two as environmental variables), but I am still getting the same error.

Am I doing something wrong or I'm not setting the environmental variables correctly? The runtime I'm using is Node.js 14x

[jonseymour]

@matthewhanson I'm getting the same error as well but I am using the ARN for a lambda layer and adding it to my lambda function, plus my lambda function I'm using to read GDAL is in nodejs not Python.

I tried to set GDAL_DATA and PROJ_LIB as environmental variables on my lambda function (I just went to Environment Variables on the UI of the lambda function and added the two as environmental variables), but I am still getting the same error.

Am I doing something wrong or I'm not setting the environmental variables correctly? The runtime I'm using is Node.js 14x

It is not inconceivable that your issue is related to a problem I have a proposed fix for - https://github.com/developmentseed/geolambda/pull/101

The issue is that the rsync step of the build process is not working, because rsync is missing from the image. This is likely failing to deliver libgdal.so into the expected place.

[jonseymour]

Having said, that I am also experiencing this very same symptom, even after fixing the Dockerfile, but only when I try to do an automated build on Circle CI.

This happens:

docker run --rm -v /home/circleci/project/ecliptic/core/lambda/python/layer/geolambda/upstream/python/lambda:/var/task -v /home/circleci/project/ecliptic/core/lambda/python/layer/geolambda/upstream/python/../lambda:/opt lambci/lambda:python3.7 lambda_function.lambda_handler '{}'

Status: Downloaded newer image for lambci/lambda:python3.7
START RequestId: e66d99e1-9a5e-1d4e-3409-467f6512600d Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': libgdal.so: cannot open shared object file: No such file or directory
END RequestId: e66d99e1-9a5e-1d4e-3409-467f6512600d
REPORT RequestId: e66d99e1-9a5e-1d4e-3409-467f6512600d  Init Duration: 125.52 ms    Duration: 4.72 ms   Billed Duration: 5 ms   Memory Size: 1536 MB    Max Memory Used: 30 MB  

{"errorType":"Runtime.ImportModuleError","errorMessage":"Unable to import module 'lambda_function': libgdal.so: cannot open shared object file: No such file or directory"}

The really annoying thing is if I run the build locally or repeat the build with Circle CI with SSH enabled, it always works without this failure, but if I then revert to the automated Circle CI build, it always fails, and I cannot work out what the key difference is. A classic Heisenbug.

update: the first docker-run command was using a -i option which is not compatible with automated execution. I will check that this option is unnecessary and amend my PR with some additional commits, the first a call to set -e to ensure the script fails if any command fails so that silent errors are not buried and the second to remove the -i option. I'll add a commit that suppresses the PROJ_LIB related noise in the last docker run step.

[bhaktatejas922]

arn:aws:lambda:us-east-1:552188055668:layer:geolambda-python:3

does this error for me too, despite setting the env variables

[petersmythe]

I was also getting this error

Unable to import module 'lambda_function': libgdal.so: cannot open shared object file: No such file or directory

when adding just the v2.1.0-python layer, but digging through the issues and PRs, especially https://github.com/developmentseed/geolambda/pull/111, led me to add the v2.1.0 layer as well (despite the note:

... plus everything in the standard GeoLambda layer. Note this is a change from v2.0.0 where both Layers needed to be included in a Lambda.

That got me past the error in this issue.