CDCgov / trusted-intermediary

Bringing together healthcare providers by reducing the connection burden.

Apache License 2.0

11 stars 5 forks source link

Spike: Confirm zero downtime deployments #1163

Closed scleary1cs closed 2 months ago

scleary1cs commented 3 months ago

Backlog Task

Ensure that our current setup supports zero downtime deployments.

If not, then we may need to prioritize "Blue/green Deployment Strategy #157" or create a new story to capture an alternative way to achieve this.

Completion Criteria

[x] Determine if we already have zero downtime with TI and ingestion service (including main app and polling function. Polling function is likely not zero downtime (there's a gap between resource creation and function creation) but that may be fine because it runs intermittently). Currently do not support zero downtime.
[x] Make new ticket looking into Azure Deployment Slots to achieve zero downtime over blue-green
- [x] If above is not true, investigate ways to achieve zero downtime through the execution of story #157

Tasks

[ ] What tasks need to be done to fulfill the completion criteria?
[ ] Ensure that the database and it's migration are accounted for in this process.

Other Notes

If yes, we might want to pull "Blue/green Deployment Strategy #157" into Production Stage 3, from Production Stage 4.

scleary1cs commented 2 months ago

Taken from SLA:

Platform Availability: -The PRIME ReportStream hosting infrastructure provides high availability with a goal of between 99.5% and 99.95% uptime (Azure dependent) for infrastructure in the production environment.
-If the downtime exceeds an hour, the entire span of time including the initial hour will be considered downtime. -Scheduled maintenance or service upgrades will not be considered by Users as Unavailable. See System Maintenance section. -Any error or incident caused by the User, or by an act of force majeure, will not be considered by the User as Unavailable.

scleary1cs commented 2 months ago

System Maintenance: CDC will be entitled to conduct system maintenance or to make system improvements on PRIME ReportStream and any normal, planned and scheduled downtime associated with this maintenance or improvement will not count against PRIME ReportStream availability. CDC will take reasonable action to notify Users of any change in the scheduled maintenance window. There are times that CDC will need to perform unscheduled maintenance that will not have been previously planned or scheduled but will require prompt attention. For unpredictable maintenance, CDC will take reasonable action to adhere to Target Time to Notify.

scleary1cs commented 2 months ago

Maintenance Window means the scheduled, recurring timeframe during which PRIME ReportStream is within downtime between 8:00 a.m. (08:00) and 9:00 a.m. (09:00) (Eastern Time) on Sundays.

scleary1cs commented 2 months ago

Operational Hours means the hours between 8:00 a.m. (08:00) and 58:00 p.m. (1720:00) (Eastern Time) every Monday through Friday, except for federal holidays or any other closure of the federal government.

scleary1cs commented 2 months ago

saquino0827 commented 2 months ago

We do not do Zero downtime deployment, we need a solution such as Azure Deployment Slots
We need to improve the timing of database migration vs deployment of our code and the availability of the database.

jcrichlake commented 2 months ago

We've written a new card under #1220. This will be our implementation for zero downtime deploys. Card #157 can be closed or deleted. @scleary1cs