amazonlinux

:warning: This now lives in https://github.com/lambgeo/docker-lambda

Create an AWS lambda like docker images and lambda layer with python 3 and GDAL.

Inspired from developmentseed/geolambda and mojodna/lambda-layer-rasterio.

Dockers images

GDAL Versions - Based on lambci/lambda-base:build

• 3.0.3 (16 Jan. 2020) / remotepixel/amazonlinux:gdal3.0
• 2.4.4 (16 Jan. 2020) / remotepixel/amazonlinux:gdal2.4

Python - Based on lambci/lambda:build-python*

• 3.0

  • remotepixel/amazonlinux:gdal3.0-py3.7

• 2.4

  • remotepixel/amazonlinux:gdal2.4-py3.7

Content: GDAL Libs and python with numpy and cython

Available drivers

Create a Lambda package

You can use the docker container to either build a full package (you provide all the libraries) or adapt for the use of AWS Lambda layer.

1. Create full package (see /examples/package)

This is like we used to do before (with remotepixel/amazonlinux-gdal images)

• dockerfile

  
  # Here we create a package (like previously with amazonlambda-gdal)
  # we use the "-build" image because we don't want need pre-installed libaries
  FROM remotepixel/amazonlinux:gdal3.0-py3.7

ENV PACKAGE_PREFIX=/var/task

COPY handler.py ${PACKAGE_PREFIX}/handler.py RUN pip install numpy rasterio mercantile --no-binary :all: -t ${PACKAGE_PREFIX}/

- package.sh
```bash
#!/bin/bash
echo "-----------------------"
echo "Creating lambda package"
echo "-----------------------"
echo "Remove lambda python packages"
rm -rdf $PACKAGE_PREFIX/boto3/ \
  && rm -rdf $PACKAGE_PREFIX/botocore/ \
  && rm -rdf $PACKAGE_PREFIX/docutils/ \
  && rm -rdf $PACKAGE_PREFIX/dateutil/ \
  && rm -rdf $PACKAGE_PREFIX/jmespath/ \
  && rm -rdf $PACKAGE_PREFIX/s3transfer/ \
  && rm -rdf $PACKAGE_PREFIX/numpy/doc/

echo "Strip shared libraries"
cd $PREFIX && find lib -name \*.so\* -exec strip {} \;

echo "Create archive"
cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *
cd $PREFIX && zip -r9q --symlinks /tmp/package.zip lib/*.so* share bin
cp /tmp/package.zip /local/package.zip

• commands

  docker build --tag package:latest .
  docker run --name lambda -w /var/task --volume $(shell pwd)/:/local -itd package:latest bash
  docker exec -it lambda bash '/local/package.sh'
  docker stop lambda
  docker rm lambda

2. Use Lambda Layer (see /examples/layer)

• dockerfile

Here we install mercantile and we add our handler method. The final package structure should be

package/
  |
  |___ handler.py  
  |___ mercantile/

Docker environment variables

A couple environment variables are set when creating the images:

• PREFIX: Path where GDAL has been installed, shoud be /opt
• GDAL_DATA: $PREFIX/share/gdal
• PROJ_LIB: $PREFIX/share/proj
• GDAL_CONFIG: $PREFIX/bin/gdal-config
• GEOS_CONFIG: $PREFIX/bin/geos-config
• PATH has been updated to add $PREFIX/bin in order to access gdal binaries

Package architecture and AWS Lambda config

Simple config

package.zip
  |
  |___ lib/      # Shared libraries (GDAL, PROJ, GEOS...)
  |___ share/    # GDAL/PROJ data directories   
  |___ rasterio/
  ....
  |___ handler.py
  |___ other python module

Lambda config

• GDAL_DATA: /var/task/share/gdal
• PROJ_LIB: /var/task/share/proj

When using Lambda layer

package.zip
  |
  |___ handler.py
  |___ other python module  

Lambda config

• GDAL_DATA: /opt/share/gdal
• PROJ_LIB: /opt/share/proj

Other variable for optimal config

• GDAL_CACHEMAX: 512
• VSI_CACHE: TRUE
• VSI_CACHE_SIZE: 536870912
• CPL_TMPDIR: "/tmp"
• GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: YES
• GDAL_HTTP_MULTIPLEX: YES
• GDAL_HTTP_VERSION: 2
• GDAL_DISABLE_READDIR_ON_OPEN: "EMPTY_DIR"
• CPL_VSIL_CURL_ALLOWED_EXTENSIONS: ".TIF,.tif,.jp2,.vrt"

Layer architecture

The AWS Layer created within this repository have this architecture:

layer.zip
  |
  |___ bin/      # Binaries
  |___ lib/      # Shared libraries (GDAL, PROJ, GEOS...)
  |___ share/    # GDAL/PROJ data directories   
  |___ python/