Workaround - Graceful Shutdown of ECS Tasks

User Story - Business Need

Minimize the risk of data loss and ensure the continuity of the application's services during the task termination process. By implementing a deregulation delay of 25 seconds we should be able to ensure the currently running Fargate tasks are not receiving new tasks when we receive the SIGKILL, so we would effectively no longer terminate in the middle of task execution. This does not gracefully shut down the running tasks. The end-goal is to provide a more reliable and stable user experience. This will ultimately improve customer satisfaction and trust in the application.

User Story(ies)

As a DevOps Engineer
I want tasks to not receive work when they shouldn't So that we avoid losing data when tasks are terminated

Additional Info and Resources

Engineering Checklist

[ ] Ensure we have the option to set Deregistration for Fargate
[ ] Set an appropriate Deregistration delay for our tasks
[ ] Ensure new tasks are fully operational before the deregistration delay finishes

Acceptance Criteria

Deregistration delay is setup and we do not see new requests on those deregistered tasks
The new tasks should start receiving requests before the old tasks are terminated

Additional Notes

All the documentation seen revolved around ECS, but we use Fargate within ECS. First thing should be making sure we have that level of control over Fargate. Also, ensure an understanding of how Fargate shuts down. The graceful shutdown doc was specific to ECS and we need to make sure it's applicable to Fargate as well.

QA Considerations

For QA to populate. Leave blank if QA is not applicable on this ticket.

Out of Scope

Non-ECS related optimizations

department-of-veterans-affairs / notification-api