Esri / arcgis-python-api

Documentation and samples for ArcGIS API for Python
https://developers.arcgis.com/python/
Apache License 2.0
1.87k stars 1.1k forks source link

Minimal-dependency version for serverless deployment #1555

Closed cgpeltier closed 1 year ago

cgpeltier commented 1 year ago

Is your feature request related to a problem? Please describe. I would like to use the ArcGIS Python API with AWS Lambda Functions. However, the arcgis library with all dependencies is very large, and over the size limit for deployment within a lambda function or as a lambda layer. There have been some threads on the Esri Community site about this, but it would be nice to get an off-the-shelf solution; maybe an official lambda layer arn or a minimal-dependency version of the library for serverless usage.

Our serverless use case is pretty straightforward - just updating an AGOL feature layer from an SEDF.

nanaeaubry commented 1 year ago

@jtroe Is there something like this at the moment or in the works?

jtroe commented 1 year ago

Our current recommendation is to use a Container, as it supports a much higher deployment size. See this readme for more information.

crackernutter commented 1 year ago

@cgpeltier I was able to figure out the minimum dependencies through some trial and error for v 2.0.1 (min dependencies might change with different versions though) and was able use the library in Lambda. I still had to use S3 to host the zip file with the arcgis packages, but the size was hovering around 50 MB.

I have a .bat file that creates the environment and installs everything needed. All you need is to zip the python folder afterwards, deploy to S3, and add it as a layer to your lambda function.

_conda create --name lambda -y CALL activate lambda conda install python=3.9 -y CALL mkdir "C:\Users\jay\AppData\Local\ESRI\conda\envs\lambda\python" pip install -r arcgismindeps.txt --platform manylinux2014_x8664 --target "C:\Users\jay\AppData\Local\ESRI\conda\envs\lambda\python" --implementation cp --python 3.9 --only-binary=:all: pip install arcgis --no-deps --target "C:\Users\jay s gregory\AppData\Local\ESRI\conda\envs\lambda\python"

The arcgismindeps.txt has: ujson requests pandas six requests_toolbelt requests_ntlm ntlm_auth cachetools lxml requests_oauthlib geomet

hildermesmedeiros commented 1 year ago

@crackernutter For your future self happiness, I would recommend pinning yourarcgismindeps.txt (requirements.txt).

achapkowski commented 1 year ago

@crackernutter glad it's working. If you use an Image and register it with AWS, you can do a full install.