Fixed broken python tests

[difu]

Could not execute python tests.

• fixed wrong filename in load_source
• wrong function name when calling hasattr

[denisahearn]

@difu Were you able to get the tests to run successfully? I made the changes to test_lambda.py that you made in your pull request, and then ran docker-compose run testpackage36, but got the following error:

$ docker-compose run testpackage36
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/geolambda, inifile:
collected 0 items / 1 errors

====================================================================================== ERRORS ======================================================================================
_______________________________________________________________________ ERROR collecting test/test_lambda.py _______________________________________________________________________
ImportError while importing test module '/home/geolambda/test/test_lambda.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/test_lambda.py:4: in <module>
    lambda_handler = imp.load_source('lambda_handler', 'lambda/lambda_function.py')
/usr/lib64/python3.6/imp.py:172: in load_source
    module = _load(spec)
lambda/lambda_function.py:12: in <module>
    from osgeo import gdal
E   ModuleNotFoundError: No module named 'osgeo'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.16 seconds ==============================================================================

I am using the code from https://github.com/developmentseed/geolambda/tree/master/geolambda-seed, plus your change to test_lambda.py. I built my docker image using developmentseed/geolambda:full, and have the following files in my lambda/lib folder:

11:07 ~/development/geolambda-seed/lambda/lib $ ls -al
total 113888
drwxr-xr-x  17 denisahearn  staff       544 Dec 16 10:41 .
drwxr-xr-x   7 denisahearn  staff       224 Dec 16 10:42 ..
-rwxr-xr-x   1 denisahearn  staff    636040 Dec 16 10:41 libdf.so.0
-rwxr-xr-x   1 denisahearn  staff  15323600 Dec 16 10:41 libgdal.so
-rwxr-xr-x   1 denisahearn  staff  15323600 Dec 16 10:41 libgdal.so.20
-rwxr-xr-x   1 denisahearn  staff  15323600 Dec 16 10:41 libgdal.so.20.3.3
-rwxr-xr-x   1 denisahearn  staff   1675976 Dec 16 10:41 libgeos-3.4.2.so
-rwxr-xr-x   1 denisahearn  staff    153960 Dec 16 10:41 libgeos_c.so.1
-rwxr-xr-x   1 denisahearn  staff   4181352 Dec 16 10:41 libhdf5.so.101
-rwxr-xr-x   1 denisahearn  staff    276040 Dec 16 10:41 libjpeg.so.62
-rwxr-xr-x   1 denisahearn  staff    276040 Dec 16 10:41 libjpeg.so.62.0.0
-rwxr-xr-x   1 denisahearn  staff    196840 Dec 16 10:41 libmfhdf.so.0
-rwxr-xr-x   1 denisahearn  staff   1618040 Dec 16 10:41 libproj.so
-rwxr-xr-x   1 denisahearn  staff   1618040 Dec 16 10:41 libproj.so.9
-rwxr-xr-x   1 denisahearn  staff   1618040 Dec 16 10:41 libproj.so.9.1.0
-rwxr-xr-x   1 denisahearn  staff     48752 Dec 16 10:41 libsz.so.2
drwxr-xr-x   3 denisahearn  staff        96 Dec 16 10:41 python3.6

If the tests do run successfully for you, any thoughts on why python cannot load the osgeo module?

Thanks, Denis

[denisahearn]

@matthewhanson Can you please provide help on my question above? I was really excited to find geolambda, and hope to be able to use it to facilitate using GDAL and Python in AWS Lambda, but so far am not having any luck getting the seed project to work locally or on AWS.

Thanks in advance, Denis

[matthewhanson]

Hello @denisahearn The osgeo error I have seen, due to a non-standard way the GDAL Python libraries are installed. I'll review the current code and see if I can reproduce this.

I'm actually updating geolambda this week and will be issuing a new version 1.0.1 with an updated GDAL, pip, and other libraries. I'm then going to see if I can publish it as a public Lambda layer, if that works you won't need to worry about building and deploying the geospatial libraries anymore and could just specify geolambda as a layer.

I'll post back here with an update this week.

[rwolniak]

@matthewhanson Any updates on version 1.0.1 or the public Lambda layer?

I was able to solve some of the build bugs by changing the urls in the Dockerfile, but am still getting the same osgeo error as @denisahearn.

[matthewhanson]

@rwolniak Yes, it's currently a work in progress but there are some changes to the develop branch.

I'm changing things a bit. Instead of targeting Python specifically, I'm removing the GDAL Python libraries and Geolambda will only contain the system geospatial libraries. This allows a more generic Lambda layer to be used across multiple runtimes (all versions Python, Node, Go, etc).

So a deployed Lambda you'll just need to install the GDAL Python lib if you want to use it, or rasterio or whatever other Pythob libs you need.

I did create a public Lambda layer, but it will not be staying around. It's already out of date as now I'm including additional compression libraries along with bumping the version of GDAL to 2.4.

But here's the ARN of the test Lambda layer, if you want to try it. This does include the Python libs. arn:aws:lambda:us-east-1:433612427488:layer:geolambda:4

I'm aiming for putting out a Geolambda 1.1 release this week and will announce the final ARN for the public Lambda layer.

[jniemin]

Awesome project you got here! I tried to add the layer with given ARN and it gives me error

You are not authorized to perform: lambda:GetLayerVersion.

Probably something how you have set access for it as public AWS numby/scipy layer can be added without problems

[matthewhanson]

@denisahearn @rwolniak I've released a 1.1.0 release (1.1.0rc1) candidate, see the master branch for this version and also the ARNs for the public layers is in the README. This is a big refactor and the base GeoLambda layer is native binaries only, it's not specific to any runtime. You can therefore use this layer with any Lambda runtime.

For Python there is an example directory containing a template for making a Python based Lambda that uses the base GeoLambda as a layer.

@jniemin Not sure what the problem was, but I've deployed new layers and tested from different accounts and all seems to work. If you encounter further problems can you open a new issue?