As part of https://github.com/navapbc/template-infra/issues/522 we implemented jobs that get triggered when a file is uploaded to S3. That is only appropriate for low-latency / low-throughput jobs like hourly/daily/weekly ETL jobs that get triggered when a file from an external data source gets uploaded to a bucket.
For high frequency, low-latency event-based jobs like processing user uploads, or real-time asynchronous tasks like submitting a claim to a legacy system, we would want the ability to have a continuously running service that processes jobs from a job queue. This ticket is for implementing that.
For each job in the configuration, the service module would:
Create an additional ECS service and override the container command
Create an SQS queue
Create a policy that can access the SQS queue and add it to the service task role
Override environment variables to add JOB_QUEUE_NAME environment variable for the service tasks
For testing purposes we probably want to add some example use case to the example app that's provided in tempate-infra and used in platform-test
Deployment notes
Note that changes to the example app aren't propagated from template-infra to platform-test automatically. You'd have to make those changes to platform-test manually.
As part of https://github.com/navapbc/template-infra/issues/522 we implemented jobs that get triggered when a file is uploaded to S3. That is only appropriate for low-latency / low-throughput jobs like hourly/daily/weekly ETL jobs that get triggered when a file from an external data source gets uploaded to a bucket.
For high frequency, low-latency event-based jobs like processing user uploads, or real-time asynchronous tasks like submitting a claim to a legacy system, we would want the ability to have a continuously running service that processes jobs from a job queue. This ticket is for implementing that.
Implementation notes
continuous_jobs.tf
or something (not sure what to name it) and include it as part the service config similar to what we did for file_upload_jobs.For each job in the configuration, the service module would:
For testing purposes we probably want to add some example use case to the example app that's provided in tempate-infra and used in platform-test
Deployment notes
Note that changes to the example app aren't propagated from template-infra to platform-test automatically. You'd have to make those changes to platform-test manually.