lambda layer support for parser (and other optional features)

[mwarkentin]

Opening an issue as requested. We currently install lambda-powertools using the SAR layer method, however it appears that the optional dependencies to use advanced features like the parser are not available in the layer. We'd like a way to include these dependencies without having to maintain our own lambda layer pipeline (which would remove most of the value of installing the main powertools dependency using the SAR application).

Some possible approaches:

• Retain a single SAR application for the powertools dependency, but include all optional dependencies
  • Concerns: overall layer size? Not sure how big these dependencies are
• Configure a pipeline to publish multiple SAR apps with various combinations of optional dependencies
  • Concerns: Operational overhead for the AWS team, potential explosion of layer combinations (especially with 5-layer limit on lambda - unless there's some news about this at reinvent :smile: )
• Offload the work to all consumers of the powertools library
  • Concerns: Lots of repeated work, managing their own pipelines to generate layers, across all users of powertools

@heitorlessa are the extras required for this functionality included in the SAR app / lambda layer? Or do those need to be handled separately? Might be worth adding a note in the docs, not sure if that would be under the parser docs or the lambda layer installation docs?

Hey @mwarkentin - the latter, handled separately as described in the parser docs.

Unsure whether it worth creating a separate Layer for that tbh given the additional operational overhead. In V2, we'd like to create each utility to be pip installable if a theory works, then we could consider multiple layers I suppose.

Happy to discuss more in a separate issue tho ;)

Originally posted by @heitorlessa in https://github.com/awslabs/aws-lambda-powertools-python/issues/118#issuecomment-729827935

[heitorlessa]

Hey @mwarkentin and @koiker - What do you think of having an additional layer with all optional dependencies installed?

Adding pydantic to the current Layer, parser's optional dependency, might break some existing customers as it increases the final Lambda package size by 70M.

Adding an additional layer with clear description that it'll contain all optional dependencies we might have (incl. in the future) allows customers who want to have an all-in experience, and it gives us room to think about 2.0 - potentially microlibs - where we can break them into categories as well as make them stand-alone.

[mwarkentin]

@heitorlessa that would work for us - do you think this would be a second layer included in the existing SAR application or a separate SAR application? 70MB is a decent size chunk I guess.. maybe those limits will be increasing soon? ;)

[koiker]

Sounds a good idea to have 2 packages: aws-powertools-core and aws-powertools-extras or aws-powertools-advanced. (I don't know, just trowing some ideas on the naming convention)

[heitorlessa]

Yep - That would be a second SAR App with an explicit name to differentiate between the one we have (similar to Rafael suggestion).

In your case, you’d need only that new Layer instead of having to add multiple layers - this obviously would come at the expense of a larger package.

If anyone in the future would like to only include certain optional dependencies they could always build their own Layer, or simply use their dependency package management tool

On Fri, 20 Nov 2020 at 22:05, Rafael M. Koike notifications@github.com wrote:

Sounds a good idea to have 2 packages: aws-powertools-core and aws-powertools-extras or aws-powertools-advanced. (I don't know, just trowing some ideas on the naming convention)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/awslabs/aws-lambda-powertools-python/issues/210#issuecomment-731406915, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZPQBBX7XQD3ADOFOOQGWLSQ3K3NANCNFSM4T3PF5OA .

[mwarkentin]

Sounds good!

[heitorlessa]

just agreed with @am29d - We're gonna work on this next @mwarkentin ;)

[mwarkentin]

Woot, excellent!

[heitorlessa]

we're hoping to get it released by EOD @mwarkentin - @am29d is on it ;)

[prasenjithaty]

@heitorlessa has this been released? I didn't see anything in the documentation.

[heitorlessa]

@am29d is on it @prasenjithaty -- we've had publishing issues and will circle back here as soon as they're fixed

[prasenjithaty]

Thanks for the update @heitorlessa. I tried creating a layer within our organizations's AWS account with just aws-lambda-powertools[pydantic] as a dependency but am hitting size limits. It appears that the unzipped package is ~307MB.

[am29d]

Hey @prasenjithaty , I have managed to create an aws-lambda-powertools-extras layer including optional dependencies and I can use the layer in my lambda funciton.

May I ask: how did you packaged the dependencies? 🤔

Now, I am not sure if I have missed any deps in my layer, so it would help me to double check and compare with your approach.

[prasenjithaty]

Hi @am29d, we use Jenkins and a Makefile to bundle the dependencies. Briefly, these are the steps:

1. Create a virtual environment python3 -m venv $(VIRTUAL_ENV);
2. Activate the virtual env and install dependencies using pip install -r requirements.txt
3. Create a tmp directory using mkdir -p ./package/tmp
4. Copy site packages

   if test -d $(VIRTUAL_ENV)/lib/$(PYTHON_VERSION); then \
   mkdir -p ./package/tmp/python/lib; \
   rsync -ar --exclude="*.dist-info" --exclude="*.pyc" --exclude="wheel*" --exclude="easy_install*" --exclude="setuptools*" $(VIRTUAL_ENV)/lib/ ./package/tmp/python/lib; \
   fi
   if test -d $(VIRTUAL_ENV)/lib64/$(PYTHON_VERSION); then \
   mkdir -p ./package/tmp/python/lib64; \
   rsync -ar --exclude="*.dist-info" --exclude="*.pyc" --exclude="wheel*" --exclude="easy_install*" --exclude="setuptools*" $(VIRTUAL_ENV)/lib64/ ./package/tmp/python/lib64; \
   fi
   chmod -R a+r ./package/tmp;

5. Terraform then archives the package/tmp directory and creates a layer.

Let me know if that is not clear.

[am29d]

Hi @prasenjithaty, thanks for sharing this is interesting!

I think some parts of the VIRTUAL_ENV can be exluded further and are not required for the runtime, but I cant tell you which one. Is there a reason you do it for lib64 as well? I have a different approach by runing pip install -t ./python/. Here are the details on how I construct the layer:

1. Build project with poetry
2. Install the wheel into a specific directory
3. export poetry dependencies for pydantic to a requirements.txt
4. pip install the requirements
5. Use SAM to package and publish the SAR app

cd aws-lambda-powertools-python
pip install poetry
poetry lock
poetry build
pip install dist/*.whl -t ../$PACKAGE_TYPE/python

if [ $PACKAGE_TYPE == "extras" ] ; then
  echo "install extras"
  poetry export -o requirements.txt --without-hashes --extras "pydantic"
  pip install -r requirements.txt -t ../$PACKAGE_TYPE/python
fi

# [...] removed some SAM specific stuff 

# zip and ship
echo "Zip and ship"
zip -rq layer.zip python
sam package --template-file template.yml --output-template-file packaged.yml --s3-bucket $LAYER_DEPLOY_BUCKET
sam publish --template packaged.yml --region $AWS_REGION

The resulting zip package is 22.4MB big and 151.9 MB uncompressed (because boto core is included with ~55MB).

We are close to release the public version of the extras layer as a public SAR app soon, so it might save you some time. Again, sorry for the delay and that you have to build and manage your own layer.

Best, Alex

[prasenjithaty]

Hi @am29d, You are right. The problem was in Step 4. Both lib and lib64 are not required to be copied. Assuming that the python runtime in Lambdas to be 64bit, I removed the line that copies the same files into ./package/tmp/python/lib and was able to create the layer. I'll test as soon as I can and report back if I find any issues.

Your steps for creating the archive looks much simpler and cleaner. I'll modify my Makefile accordingly.

Thanks.

[am29d]

Hey all, we have merged the PR containing the docs changes regarding the new powertools layer with extras dependencies. The new layer aws-lambda-powertools-python-layer-extras is already published and you can start using it.

https://serverlessrepo.aws.amazon.com/applications/eu-west-1/057560766410/aws-lambda-powertools-python-layer-extras