search

department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
92 stars 68 forks source link

2021-12-27 Content release issues #7409

Closed timcosgrove closed 2 years ago

timcosgrove commented 2 years ago

Summary

Two separate issues interfered with content release on 2021-12-27. Both were eventually resolved and content release was restored.

Github Actions failure

Github Actions experienced degraded performance on 2021-12-27. Initially this resulted in content release failing and not reporting as failed. Later, a content release became 'stuck' running. Because a running content release prevents subsequent releases from running until it is finished, this effectively blocked content release.

Neil Hastings reached out to FE Tools oncall for help (https://dsva.slack.com/archives/C0MQ281DJ/p1640615811165900) FE Tools in turn reached out to Github for support (https://dsva.slack.com/archives/CU1E4CX9U/p1640623379271800). After some time, Github was able to identify the issue and resolve it. Further, the Github Actions team is going to add a tool to allow VA OIT Ops to cancel actions in the event this kind of thing happens again.

Platform CMS followup

Nothing should be necessary for the time being. The issue did not arise due to shortcomings on our end; our team identified and reported the issue through expected channels; and the issue was resolved by another team.

GI Bill Comparison Tool URL change.

As part of normal work, the AFS Education team submitted a URL change for the GI Bill Comparison Tool. The work done accounted for the URL change within the content-build and vets-websites repos, but could not account for CMS content that links to the old URL. This triggered a significant amount of broken links in excess of 10 links, breaking content release.

Platform CMS worked with the engineers in question to help restore content release. We reverted the changes which took a significant amount of footwork due to the holidays and code freeze. We worked out a plan with the engineers in question to safely make this transition, which will need to include the CMS team coordinating an update of CMS content that links to the old URL.

Potential followup

  1. There is no established process for non-CMS teams to alert CMS that URL changes are coming. It may be that CMS is not part of consideration when planning a URL change like this. We may want to work to integrate awareness of and coordination with CMS into processes like redirect requests
timcosgrove commented 2 years ago

Ended up duplicated. Use #7410