search

serverless / serverless-python-requirements

⚡️🐍📦 Serverless plugin to bundle Python packages
MIT License
1.11k stars 290 forks source link

Store dependencies in EFS #540

Open geoffsmith opened 4 years ago

geoffsmith commented 4 years ago

It would be amazing to be able to use EFS to store large dependencies instead of layers!

One common use-case is trying to load deep learning models which depend on something like pytorch or tensorflow, libraries that are several hundred mb. It's very tricky to get them to fit into layers. Another use-case I've come across is geospatial processing depending on gdal, which again has a massive footprint.

Here's the AWS blog post where they announce lambda support for EFS: https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/

nuc-ronniehyslop commented 1 year ago

I would like to add to this request, it would be so useful to be able to set a config flag to EFS and all the requirements are added to a new or existing EFS share.

My team work with Oracle DBs a lot and the size of the oracle binaries and cx_Oracle library regularly give us packaging size issues. Layers are helpful but don't overcome the total size issue.

I'm not sure if this request sits in this plugin or if it should be raised as a feature request for the core serverless framework.