Using public ARN - No module named osgeo

[teknocreator]

Good day! I'm attempting to use public geolambda ARN for us-east-1, arn:aws:lambda:us-east-1:552188055668:layer:geolambda-dev:2 along with the example lambda_function.py you've provided in hopes I can use it for our current project. I've added the ARN above as a layer to that example function. However, when testing, I get the following error: "Unable to import module 'lambda_function': No module named osgeo". Perhaps I've missed something in the documentation, but it looked like I should just be able to use the example function and test this out of the box without having to create a new version. I'm in the US East region and I didn't have an issue when setting up the test lambda function to use the public geolambda ARN. Hopefully, I've just missed something simple. Thanks!

[vincentsarago]

Dear @teknocreator thanks for opening the issue! in https://github.com/developmentseed/geolambda/pull/51 released rencently we decided to remove gdal python bindings from the image to let the user choose the bindings of their choice (gdal, gippy or Rasterio) instead.

Please let me know if this is useful information

[teknocreator]

That's fine and it looked like that was for folks that wanted to build a new version. But, I thought from the docs that the public ARN arn:aws:lambda:us-east-1:552188055668:layer:geolambda-dev:2 was prebuilt for python and went along with the lambda_function.py example? Perhaps I read that incorrectly?

[matthewhanson]

Hi @teknocreator ,

The Lambda layer is not for Python specifically, it can be used with any Lambda runtime as it just contains native binary libraries.

For Python you'll want to take a look at the README in the Python directory, it will step you through using it for Python.

The reason for doing it this way is that once the native libraries are available in Lambda, deploying whatever Python packages you need is fairly easy, and everyone has different requirements for what Python packages are needed.

[teknocreator]

Okay, well, I'll take a look again at the README in the Python directory and see what we can do. Thanks for getting back to me and clarifying.

[robertd]

I followed the python/readme.md and I'm still getting the following error: Unable to import module 'lambda_function': No module named 'osgeo'

I've also ran official dockerhub image to create lambda-deploy.zip using docker run -v ${PWD}:/home/geolambda -t developmentseed/geolambda:1.1.0-python36 package-python.sh, and I'm still getting the same error.

Would you please consider releasing arn:aws:lambda:us-west-2:552188055668:layer:geolambda-python36:1?

[robertd]

Also, Testing deployment package section is not working for me:

[robertd]

So I've tried to unzip lambda-deploy.zip into .venv environment I created, but I'm getting different error now ImportError: No module named '_gdal'

@matthewhanson Any help would be greatly appreciated.

[matthewhanson]

Hi @robertd ,

I think the problem might be that when testing locally you also need to package up the base GeoLambda files. It's not exactly clear in the docs, but in order to test you mount the GeoLambda files to /opt and your handler and python libs at /var/lib. It looks like it can't find the GDAL library s this might be the problem.

In the top level of this repo, you will want to call:

$ docker run --rm -v $PWD:/home/geolambda \
    -it developmentseed/geolambda:1.1.0 package.sh

This is shown in the top level README under development, but it's not clear that you also need to do this for local testing.

This will put those lib files in a lambda directory. Then you build and package the Python handler part. And then you run the tests mounting both directories.

Also, I agree a Python based Lambda layer (arn:aws:lambda:us-west-2:552188055668:layer:geolambda-python36) would be useful, I'll open an issue for it and get it deployed with the next version. Note that it would be a Layer containing just the Python libraries osgeo (gdal) and rasterio, so a Lambda function would need to include both the geolambda and geolambda-python36 layers.

Reopening this issue, please report back your results and thanks!

[robertd]

Hi @matthewhanson,

That worked.

Thanks for clarifying that both geolambda and geolambda-python36 layers are needed as dependencies.

Since geolambda is already published (i.e. arn:aws:lambda:us-west-2:552188055668:layer:geolambda:1) and user would be responsible for publishing (at the moment) of geolambda-python36 layer (by uploading lambda-deploy.zip); does ZIP file structure behind geolambda-python36 needs to follow AWS convention to be a proper Lambda Layer (i.e. contents of the zip file should be in the python/ dir, etc)? I guess I can give it a try and let you know.

Thanks for considering releasing official geolambda-python36 which would really make it turnkey solution for us.

[robertd]

@matthewhanson It seems that zip dir structure needs to be nested in python/ dir. That means package-python.sh will have to be updated to nest current contents in python/.

[matthewhanson]

@robertd great, glad to hear you got it working.

Thanks also for pointing out the change when making it a layer the Python packages need to be in a python/ subdirectory, will open issue now for making the geolambda-python36 layer, but I don't have a timeline on when that will be yet.

[robertd]

@matthewhanson Also, this python/readme.md section needs updating...

Deploy the Lambda function now by zipping up the lambda directory and using the AWS CLI.

$ aws lambda update-function-code --function-name <mylambda> --zip-file fileb://lambda-deploy.zip

update-function-code assumes that function/layer is already out there. I believe this should be used instead:

aws lambda publish-layer-version --layer-name <my-lambda-layer> --zip-file fileb://lambda-deploy.zip 

Of course, after #62 is completed.

I can lend a hand if needed... just let me know.

[drzhouq]

@robertd Has anyone created a public layer for the geolambda-python36? Or we still have to create the layer ourselves? Thanks.

[NaorManna]

@robertd, Can you share the layer.zip and the python example that using gdal?

[robertd]

@drzhouq I don't think @matthewhanson has created python36 layer yet.

[robertd]

@NaorManna example and lambda-layer-deploy.zip

[robertd]

@drzhouq @NaorManna P.S. Sorry for late reply.

[drzhouq]

@robertd many thanks.

[drzhouq]

@robertd @vincentsarago I tested the two layers locally and it was successful. However, when I created those two layers and test on aws, I got this error. Not sure what's going on. Please help. (The only conclusion I have is that the lambci/lambda:python3.6 doesn't match the AWS lambda): START RequestId: 886d0fac-b54d-4254-af0e-d943bb3e8918 Version: $LATEST Unable to import module 'lambda_function': /lib64/libcrypto.so.10: version `OPENSSL_1.0.2' not found (required by /opt/lib/libcurl.so.4)

END RequestId: 886d0fac-b54d-4254-af0e-d943bb3e8918 REPORT RequestId: 886d0fac-b54d-4254-af0e-d943bb3e8918 Duration: 0.39 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 56 MB

[robertd]

@drzhouq Here is an example repo using aws-cdk: https://github.com/robertd/geolambda-python-example

Lambda function

import os
import subprocess
import sys
import logging

# set up logger
logger = logging.getLogger(__file__)
logger.setLevel(logging.DEBUG)
# commented out to avoid duplicate logs in lambda
# logger.addHandler(logging.StreamHandler())

# imports used for the example code below
from osgeo import gdal

test_filename = 'https://landsat-pds.s3.amazonaws.com/c1/L8/086/240/LC08_L1GT_086240_20180827_20180827_01_RT/LC08_L1GT_086240_20180827_20180827_01_RT_B1.TIF'

def lambda_handler(event, context=None):
    """ Lambda handler """
    logger.debug(event)

    # this try block is for testing and info only,
    # it prints out info on the the libgdal binary and paths to linked libraries
    #try:
    #    output = subprocess.check_output('readelf -d /opt/lib/libgdal.so'.split(' '))
    #    logger.info(output.decode())
    #    output = subprocess.check_output('ldd /opt/lib/libgdal.so'.split(' '))
    #    logger.info(output.decode())
    #except Exception as e:
    #    pass

    # process event payload and do something like this
    fname = event.get('filename', test_filename)
    fname = fname.replace('s3://', '/vsis3/')
    # open and return metadata
    ds = gdal.Open(fname)
    band = ds.GetRasterBand(1)
    stats = band.GetStatistics(0, 1)

    return stats

if __name__ == "__main__":
    """ Test lambda_handler """
    event = {'filename': test_filename}
    stats = lambda_handler(event)

[drzhouq]

@robertd. Thanks. I used the exact sample code. I build both layers with this repo. The error was about missing openssl 1.0.2, which is supposed to be provided by the lambda. I tested the layers and sample codes locally without any issue.

[robertd]

@drzhouq what happens when you run cdk deploy in the project I gave you? That should create the lambda stack with layers and everything (that’s the screenshot I pasted). Let me know.

[drzhouq]

@robertd Thanks for sharing. I cloned your repo and deployed the lambda layer and function. It works (well, I made a little bit change to use the east region public layer). Then I went back to use the public layer, my original function also works now. To summary, with the public gdal layer and the python layer (yours or the one I created), it works. However, if I use the layer file (the gdal base) created in this repo, I still got the openssl error. I will create another issue for that. Again, many thanks for your help.

[robertd]

@drzhouq 👍

[robertd]

@drzhouq which branch did you use to create base gdal layer? Development or master?

[drzhouq]

The master branch, @robertd As I said earlier, the local test was successful (mount the base layer to /opt and the python layer /opt/python). The only explanation for why the local test is successful, but the same layer wasn't successful on aws lambda is that somehow lambci/lambda:python3.6 don't match the lambda environment. That theory cannot explain why the identical public layer hosted by developmentseed works. If you have time, you can try to upload a locally built base layer and see what happens.

[robertd]

It seems that underlying lambci Docker layer is probably where the "problem" is. Similar issue: https://github.com/lambci/docker-lambda/issues/179

[drzhouq]

yes, you are right, @robertd. The openssl on aws lamba is "OpenSSL 1.0.1k-fips 8 Jan 2015" (via

import subprocess
def lambda_handler(event, context=None):
    logger.debug(subprocess.check_output(['openssl', 'version']).decode("utf-8"))

, and the version in the is "OpenSSL 1.0.2k-fips 26 Jan 2017" (via command docker run --rm --entrypoint openssl lambci/lambda:python3.6 version).

[robertd]

@drzhouq After further investigation... lambci/lambda:build-provided has libcrypto.so.10 in /usr/lib64 dir, where AWS Lambda runtime doesn't.

lambci/lambda:build-provided

bash-4.2# ls -la /usr/lib64/libcry*
lrwxrwxrwx 1 root root      19 May 15 14:19 /usr/lib64/libcrypto.so -> libcrypto.so.1.0.2k
lrwxrwxrwx 1 root root      19 May  7 01:02 /usr/lib64/libcrypto.so.10 -> libcrypto.so.1.0.2k
-rwxr-xr-x 1 root root 2468552 Apr  3 23:54 /usr/lib64/libcrypto.so.1.0.2k

AWS Lambda runtime

root root    14144 Feb  1  2017 libxcb-xf86dri.so.0.0.0
lrwxrwxrwx  1 root root       22 May 14 21:10 libxcb-xfixes.so.0 -> libxcb-xfixes.so.0.0.0
-rwxr-xr-x  1 root root    25264 Feb  1  2017 libxcb-xfixes.so.0.0.0
lrwxrwxrwx  1 root root       24 May 14 21:10 libxcb-xinerama.so.0 -> libxcb-xinerama.so.0.0.0
-rwxr-xr-x  1 root root     8752 Feb  1  2017 libxcb-xinerama.so.0.0.0
lrwxrwxrwx  1 root root       19 May 14 21:10 libxcb-xkb.so.1 -> libxcb-xkb.so.1.0.0
-rwxr-xr-x  1 root root   104328 Feb  1  2017 libxcb-xkb.so.1.0.0
lrwxrwxrwx  1 root root       24 May 14 21:10 libxcb-xselinux.so.0 -> libxcb-xselinux.so.0.0.0
-rwxr-xr-x  1 root root    28736 Feb  1  2017 libxcb-xselinux.so.0.0.0
lrwxrwxrwx  1 root root       21 May 14 21:10 libxcb-xtest.so.0 -> libxcb-xtest.so.0.0.0
-rwxr-xr-x  1 root root     7016 Feb  1  2017 libxcb-xtest.so.0.0.0
lrwxrwxrwx  1 root root       20 May 14 21:10 libxcb-xvmc.so.0 -> libxcb-xvmc.so.0.0.0
-rwxr-xr-x  1 root root    12840 Feb  1  2017 libxcb-xvmc.so.0.0.0
lrwxrwxrwx  1 root root       18 May 14 21:10 libxcb-xv.so.0 -> libxcb-xv.so.0.0.0
-rwxr-xr-x  1 root root    25048 Feb  1  2017 libxcb-xv.so.0.0.0
lrwxrwxrwx  1 root root       22 May 14 21:10 libXcomposite.so.1 -> libXcomposite.so.1.0.0
-rwxr-xr-x  1 root root     9592 Feb 27  2013 libXcomposite.so.1.0.0
lrwxrwxrwx  1 root root       19 May 14 21:10 libXdamage.so.1 -> libXdamage.so.1.1.0
-rwxr-xr-x  1 root root     8816 Feb 27  2013 libXdamage.so.1.1.0
lrwxrwxrwx  1 root root       16 May 14 21:10 libXext.so.6 -> libXext.so.6.4.0
-rwxr-xr-x  1 root root    74000 Mar 12  2015 libXext.so.6.4.0
lrwxrwxrwx  1 root root       18 May 14 21:10 libXfixes.so.3 -> libXfixes.so.3.1.0
-rwxr-xr-x  1 root root    20336 Nov 20  2014 libXfixes.so.3.1.0
lrwxrwxrwx  1 root root       17 May 14 21:10 libXfont.so.1 -> libXfont.so.1.4.1
-rwxr-xr-x  1 root root   212592 Sep 11  2015 libXfont.so.1.4.1
lrwxrwxrwx  1 root root       14 May 14 21:10 libXi.so.6 -> libXi.so.6.1.0
-rwxr-xr-x  1 root root    61008 Nov 20  2014 libXi.so.6.1.0
lrwxrwxrwx  1 root root       16 May 14 21:10 libxml2.so.2 -> libxml2.so.2.9.1
-rwxr-xr-x  1 root root  1469288 Aug 23  2018 libxml2.so.2.9.1
lrwxrwxrwx  1 root root       19 May 14 21:10 libXrender.so.1 -> libXrender.so.1.3.0
-rwxr-xr-x  1 root root    38224 Nov 20  2014 libXrender.so.1.3.0
drwxr-xr-x  2 root root     4096 Mar  4  2015 libxslt-plugins
lrwxrwxrwx  1 root root       17 May 14 21:10 libxslt.so.1 -> libxslt.so.1.1.28
-rwxr-xr-x  1 root root   252272 Mar  4  2015 libxslt.so.1.1.28
lrwxrwxrwx  1 root root       14 May 14 21:10 libXt.so.6 -> libXt.so.6.0.0
-rwxr-xr-x  1 root root   410296 Nov 20  2014 libXt.so.6.0.0
lrwxrwxrwx  1 root root       16 May 14 21:10 libXtst.so.6 -> libXtst.so.6.1.0
-rwxr-xr-x  1 root root    21944 Mar 12  2015 libXtst.so.6.1.0
lrwxrwxrwx  1 root root       19 May 14 21:10 libXxf86vm.so.1 -> libXxf86vm.so.1.0.0
-rwxr-xr-x  1 root root    19256 Nov 20  2014 libXxf86vm.so.1.0.0
drwxr-xr-x  2 root root     4096 Apr 17 09:08 lksctp-tools
drwxr-xr-x  3 root root     4096 Apr 17 09:07 lua
drwxr-xr-x  4 root root     4096 Apr 17 09:08 nss
drwxr-xr-x  3 root root     4096 Apr 17 09:07 openssl
drwxr-xr-x  2 root root     4096 Apr 17 09:08 p11-kit
lrwxrwxrwx  1 root root       19 May 14 21:10 p11-kit-proxy.so -> libp11-kit.so.0.0.0
drwxr-xr-x 29 root root     4096 Apr 17 09:07 perl5
drwxr-xr-x  2 root root     4096 Apr 17 09:08 pkcs11
drwxr-xr-x  2 root root     4096 Apr 17 09:07 pkgconfig
dr-xr-xr-x  5 root root     4096 Apr 17 09:07 pm-utils
drwxr-xr-x 27 root root    20480 Apr 17 09:07 python2.7
drwxr-xr-x 33 root root     4096 Apr 17 09:07 python3.4
drwxr-xr-x  2 root root     4096 Apr 17 09:07 sasl2
dr-xr-xr-x  2 root root     4096 Jan  6  2012 sse2
drwxr-xr-x  2 root root     4096 Apr 17 09:08 tc
dr-xr-xr-x  2 root root     4096 Jan  6  2012 tls
dr-xr-xr-x  2 root root     4096 Jan  6  2012 X11

@matthewhanson Any idea how to port that over? I've tried downgrading openssl to 1.0.1k (https://forums.aws.amazon.com/thread.jspa?messageID=823346&tstart=0) but that didn't work. I've also tried updating package.sh to copy over libcrypto dependencies to /opt/lib64, but that didn't work either. Any ideas?

# make deployment directory and add lambda handler
mkdir -p $DEPLOY_DIR/lib
mkdir -p $DEPLOY_DIR/lib64

# copy libs
cp -P ${PREFIX}/lib/*.so* $DEPLOY_DIR/lib/
cp -P ${PREFIX}/lib64/libjpeg*.so* $DEPLOY_DIR/lib/
cp -P /usr/lib64/libcrypto*.so* $DEPLOY_DIR/lib64/

[robertd]

@drzhouq You're absolutely right.

I've pulled amazonlinux:1 and openssl on it is 1.0.2k.

~/robertd docker run --rm -it amazonlinux:1 /bin/bash
bash-4.2# uname -a
Linux 6271c1cfa5f7 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
bash-4.2# python --version
Python 2.7.16
bash-4.2# python
Python 2.7.16 (default, Mar 18 2019, 18:38:44)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output(['openssl', 'version']).decode("utf-8")
u'OpenSSL 1.0.2k-fips  26 Jan 2017\n'

Same as in lambci/lambda:build-provided

~/robertd docker run --rm -it lambci/lambda:build-provided /bin/bash                                                            125 ↵  10:21:36
bash-4.2# uname -a
Linux 5c81ffc6ecdb 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
bash-4.2# python --version
Python 2.7.16
bash-4.2# python
Python 2.7.16 (default, Mar 18 2019, 18:38:44)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.check_output(['openssl', 'version']).decode("utf-8")
u'OpenSSL 1.0.2k-fips  26 Jan 2017\n'

AWS Lambda python3.6 runtime is has older version:

START RequestId: a26afc85-215b-4529-b329-5aac9d9e7c6c Version: $LATEST
[DEBUG] 2019-05-28T16:36:17.28Z a26afc85-215b-4529-b329-5aac9d9e7c6c    {}
[DEBUG] 2019-05-28T16:36:17.141Z    a26afc85-215b-4529-b329-5aac9d9e7c6c    OpenSSL 1.0.1k-fips 8 Jan 2015

END RequestId: a26afc85-215b-4529-b329-5aac9d9e7c6c
REPORT RequestId: a26afc85-215b-4529-b329-5aac9d9e7c6c  Duration: 113.20 ms Billed Duration: 200 ms     Memory Size: 128 MB Max Memory Used: 50 MB  

Where AWS Lambda python3.7 runtime has the correct version but gdal doesn't run for some reason.

START RequestId: ad9a8b8b-fbcc-478a-b5f6-e688575edc50 Version: $LATEST
[DEBUG] 2019-05-28T16:43:48.105Z    ad9a8b8b-fbcc-478a-b5f6-e688575edc50    {}

[DEBUG] 2019-05-28T16:43:48.219Z    ad9a8b8b-fbcc-478a-b5f6-e688575edc50    OpenSSL 1.0.2k-fips  26 Jan 2017

END RequestId: ad9a8b8b-fbcc-478a-b5f6-e688575edc50
REPORT RequestId: ad9a8b8b-fbcc-478a-b5f6-e688575edc50  Duration: 135.35 ms Billed Duration: 200 ms     Memory Size: 128 MB Max Memory Used: 55 MB  

Error when using python3.7 as lambda runtime.

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named '_gdal'

FWIW AWS blog noted that they'll be moving to the newer AmazonLinux 2018.03 runtime in June. https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/ . This explains why geolambda works locally with lambci/lambda:build-provided (python36) but not on AWS because lambda is using older runtime at the moment. Maybe we just sit and wait until then since it's around the corner.

[drzhouq]

@robertd Since AWS python3.7 has the correct openssl, we can rebuild our python layer under 3.7. I did that and it worked. Thanks for your help, I now have two layers working correctly under python3.7 on lambda. The package-python.sh is

#!/bin/bash

# directory used for deployment
export DEPLOY_DIR=lambda

# Get Python version
PYVERSION=$(cat /root/.pyenv/version)
MAJOR=${PYVERSION%%.*}
MINOR=${PYVERSION#*.}
PYVER=${PYVERSION%%.*}.${MINOR%%.*}
#note changed the hardcoded version
PYPATH=/root/.pyenv/versions/$PYVERSION/lib/python${PYVER}/site-packages/

echo Creating deploy package for Python $PYVERSION

EXCLUDE="boto3* botocore* pip* docutils* *.pyc setuptools* wheel* coverage* testfixtures* mock* *.egg-info *.dist-info __pycache__ easy_install.py"

EXCLUDES=()
for E in ${EXCLUDE}
do
    EXCLUDES+=("--exclude ${E} ")
done
# note, put the files under the python folder 
rsync -ax $PYPATH/ $DEPLOY_DIR/python ${EXCLUDES[@]}

# zip up deploy package
cd $DEPLOY_DIR
zip -ruq ../lambda-deploy.zip ./

And to build the image, docker build . --build-arg VERSION=${VERSION} --build-arg PYVERSION=3.7.3 -t mytest to generate the python layer zip file docker run --rm -v ${PWD}:/home/geolambda -t mytest package-python.sh. Please note the lambda folder contains no example lambda_function.py file. Though the file is not harmful, but it is better not to include it in the layer file.

[robertd]

@drzhouq Very cool. 👍

Btw... are you sure that you're building this in develop branch? Your package-python.sh doesn't look like it's from develop branch because I've already addressed these issues in this commit: a9b777876751bc524b87d68e596b93015b2249fb

#!/bin/bash

# directory used for development
export DEPLOY_DIR=lambda

# Get Python version
PYVERSION=$(cat /root/.pyenv/version)
MAJOR=${PYVERSION%%.*}
MINOR=${PYVERSION#*.}
PYVER=${PYVERSION%%.*}${MINOR%%.*}
PYPATH=/root/.pyenv/versions/$PYVERSION/lib/python3.6/site-packages/

echo Creating deploy package for Python $PYVERSION

EXCLUDE="boto3* botocore* pip* docutils* *.pyc setuptools* wheel* coverage* testfixtures* mock* *.egg-info *.dist-info __pycache__ easy_install.py"

EXCLUDES=()
for E in ${EXCLUDE}
do
    EXCLUDES+=("--exclude ${E} ")
done

rsync -ax $PYPATH/ $DEPLOY_DIR/ ${EXCLUDES[@]}

# prepare dir for lambda layer deployment - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
cp -r $DEPLOY_DIR ./python
rm ./python/lambda_function.py

# zip up deploy package
zip -ruq lambda-layer-deploy.zip ./python

# cleanup
rm -rf ./python

Perhaps you can issue a PR with your updates to package-python.sh

edit: I see what you did there. I'll issue a PR and cc you to confirm this works.

[kylebarron]

would need to include both the geolambda and geolambda-python36 layers

I don't think that was made clear in the README... I was trying just the python layer and it couldn't find the libraries without the main geolambda layer