Postmortem: 5/8/2024 Module Installation Broke Automated Production Deploy

gracekretschmer-metrostar commented 2 months ago

Background

On 5/8/2024, CMS prod went offline due to a module installation. The folder required for installing the translations of the admin UI were not there and caused the deploy to fail and take prod CMS offline.

User Story or Problem Statement

Drupal developers need to be able to install contributed modules when necessary to add or maintain functionality needed by editors to ensure that content is up-to-date and complete. When new modules are enabled within Drupal, they write a file to the docroot/sites/default/files/translations directory that contains translations of the UI elements for that module. If the directory is not present or doesn't have correct permissions there is an error generated which fails the deployment.

Reference Links

https://www.drupal.org/docs/administering-a-drupal-site/security-in-drupal/securing-file-permissions-and-ownership

Affected users and stakeholders

CMS_editors
Product teams working within CMS

Hypothesis

Ensure that the Ansible creates the folder that the translations file config will go into
Would it make sense to have the Apache user manage all the folders/files that the CMS uses to release new modules?
Make CMS the owner and apache a user in the group.

Solution

Add scripting to the deployment process to fix the file permissions and to check if the docroot/sites/default/files/translations directory exists and create it if it does not. This could extend the deployment time but will result in a more consistent and reliable deployment process.

Acceptance Criteria

[x] The root cause of why prod CMS went offline on 5/8/2024.
[x] A solution is implemented that resolves prod CMS from going offline during daily deploy.
[x] Opportunities to proactively prevent prod CMS from going offline in the future.
[ ] Postmortem report submitted to Github repo.

gracekretschmer-metrostar commented 2 months ago

Postmortem retro board: https://easyretro.io/publicboard/NqWKfJWUr2dFRoCJjX2ybXcRyws2/eedbcd70-6144-4945-884c-1ade55afb56b

gracekretschmer-metrostar commented 1 month ago

@7hunderbird how is the postmortem going? Do you need any support?

7hunderbird commented 1 month ago

I've found a ticket where I had captured the timeline here: https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18085
- I've closed #18085 because the timeline is captured in the va.gov-team-sensitive pull-request
That timeline is part of the PR draft I'm working on here: https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/pull/1724

gracekretschmer-metrostar commented 1 month ago

Tyler: right now, have done 50% done with the report and then we can determine if we need to have a postmortem meeting.

7hunderbird commented 3 weeks ago

I have submitted https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/pull/1724 for review from @timcosgrove and @gracekretschmer-metrostar

gracekretschmer-metrostar commented 3 weeks ago

Need review from @EWashb.

department-of-veterans-affairs / va.gov-cms