wri / gfw-raster-analysis-lambda

GFW Raster Analysis in AWS Lambda
8 stars 4 forks source link

Prototype with Step Function #11

Open jterry64 opened 4 years ago

jterry64 commented 4 years ago

Create a prototype using this lambda function with Step Function's Map State to see if it can work as intended (calling 100s-1000s in parallel with high performance) and to see if Step Function could work to connect the components of this service.

It seems like the Map State feature was only released a week ago, so this will also let us see if it's ready to use at our scale yet.

jterry64 commented 4 years ago

It works, although we need to do some workarounds with S3 (or possibly DynamoDB) to deal with the number of geometries we want to process. It also is not incredibly stable at this moment, and usually will just fail with an internal error if you try more than 50 lambdas at once.

Currently an open ticket on AWS support center discussing these issues.