search

weather-gov / weather.gov

weather.gov 2.0
Other
329 stars 9 forks source link

Update deployment process to be all-or-nothing #2063

Open greg-does-weather opened 1 week ago

greg-does-weather commented 1 week ago

Description

Currently when we deploy, it's a three-step process:

  1. deploy interop layer
  2. deploy Drupal
  3. start post-deploy task

This could result in a period of inconsistency where the interop layer is updated but Drupal is not, and that could lead to users seeing unpredictable results. Additionally, if the interop layer deployment succeeds but the Drupal deployment fails, we are left in that inconsistent state indefinitely.

Both of these potential issues can be mitigated by modifying our deployment process to push to temporary apps and then remapping our routes to those new apps as a very last step. The rolling strategy we currently use is great for a single app, but keeping multiple apps released in sync requires a little more careful nuanced approach, I think.

Alternatively, the interop layer could be treated as an entirely separate product and deployed separately on its own timeline, but that creates its own logistical problems.

Acceptance criteria

colinmurphy01 commented 6 days ago

this feels like something we should fix prior to the next release?

greg-does-weather commented 6 days ago

I don't think so. Before IOC is probably fine. But I'd defer to @jamestranovich-noaa because he knows more about infra than I do!

jamestranovich-noaa commented 2 days ago

Yep, we do not need to address this before IOC. I think we are OK -- if the interop layer fails via cf push then the rest of the steps should fail (but please correct me if I'm wrong here @greg-does-weather). We wait for up to 5 minutes for the app to start. If the Drupal app fails, then we can restart the deploy. But yes, I agree this is a concern and we should address this down the road.

That aside, there might be a larger conversation here about how to seamlessly manage releases -- the cloud.gov recommended route is blue-green releases which requires essentially two production environments, but again, we can revisit before IOC.