search

awslabs / aws-solutions-constructs

The AWS Solutions Constructs Library is an open-source extension of the AWS Cloud Development Kit (AWS CDK) that provides multi-service, well-architected patterns for quickly defining solutions
https://docs.aws.amazon.com/solutions/latest/constructs/
Apache License 2.0
1.23k stars 247 forks source link

New Pattern: lambda-comprehend #97

Open knihit opened 3 years ago

knihit commented 3 years ago

Based on which feature and service of comprehend you plan to use, comprehend requires permission policy setup so that lambda functions can access them. Some of the API endpoints in Comprehend also require an S3 bucket to be setup for input and ouput. Specifically in case of topic modeling, the outputs are written to the S3 bucket after about 25-30 mins. Which would mean that we need a lambda function bound to an S3 notification to be notified once the data becomes available. Hence the need for a higher level construct to provide the plumbing required to integrate lambda with comprehend services.

Use Case

Comprehend is one of key NLP services for AWS AI/ML services. For my use-case, I ingest text data and subject that to various comprehend services to derives inferences. This helps understand public perception and our customers have a better understanding of customer feedback

Proposed Solution

The proposed solution is that

  1. Provide the flexibility to the user to specify which Amazon Comprehend services does he plan to integrate with
  2. Build the service role permissions required to access lambda
  3. If the service selected requires S3 bucket, create the bucket too and update the bucket permissions as required by comprehend to access it
  4. If the NLP output from comprehend is to be put in an S3 bucket (non-realtime inference), build the plumbing required to generate an S3 notification to a lambda function.

Other

I currently use this pattern in https://aws.amazon.com/solutions/implementations/discovering-hot-topics-using-machine-learning/

This is a :rocket: Feature Request

hnishar commented 3 years ago

Overall the pattern seems like a good candidate, the only feedback is #4 can make use of the existing aws-s3-lambda pattern under the hood

knihit commented 3 years ago

In general I believe yes we should be able to use existing pattern, there would be some additional setup on top of that construct to allow comprehend to write data to the S3 bucket.