We need to terraform the following pieces of infrastructure to support the data ingestion flow, ensuring that the app is maintainable in the long term and can be fed updated data as needed.
Acceptance Criteria
An s3 bucket is provisioned with two directories application_questions and institution_data
A lambda function that is created based on the ingest-instutitons.py file in the utilities folder of the code base
The lambda function should have a trigger that happens when the there is a new filed added to institution_data/ that have the suffix .csv
A SQS message queue that the above lambda function can write messages to, called process-institutions
A lambda function that is created based on the generate-descriptions-and-images.py file
The lambda should trigger by new messages in the queue
A DynamoDB named institutions
Amazon Bedrock with models Titan Image Generator G1, Titan Text G1 - Lite, and
Jurassic-2 Mid enabled
A lambda function that is created based on the ingest-applications.py file in the utilities folder of the codebase
A trigger on the lambda function that hits every time a files is added to application_questions/ that have the suffix .csv in the s3 bucket from step 1
A s3 bucket called swift-institution-images that the lambda from point 5 can write to
Description
We need to terraform the following pieces of infrastructure to support the data ingestion flow, ensuring that the app is maintainable in the long term and can be fed updated data as needed.
Acceptance Criteria
application_questionsandinstitution_dataingest-instutitons.pyfile in the utilities folder of the code basegenerate-descriptions-and-images.pyfileAdditional Notes
Definition of Done
Definition of Done