department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
99 stars 69 forks source link

Document and Synthesize Tugboat Upgrade Information #19310

Closed gracekretschmer-metrostar closed 3 weeks ago

gracekretschmer-metrostar commented 1 month ago

User Story or Problem Statement

To help towards the planning of the Tugboat upgrade, the CMS team needs to take rigorous documentation to help towards the planning phase of the upgrade.

Description or Additional Context

Tugboat and Docker will need to be upgraded because the current file system used by both is being deprecated. The CMS team is meeting with Tugboat to understand the impact and scope of this work, so we want to document those conversations and our findings, to help towards the planning of this work.

Steps for Implementation

Acceptance Criteria

anantais commented 1 month ago

Notes from the meeting on 09/25:

We went over a new idea for the update:

set up a new server to graduallly move our previews over a period of time

this will result in close to no down time create the new server and add it to tugboat as a worker the current server is both a manager and worker worker would be configured to run previews in order to do this - add a new server (using docker swarm) that is just a worker while keeping the current server as a manager no new previews would be created in current server we would need to wait 24 hours for previews to rebuild in worker server then we can have full control of when we rebuild previews adding this new server would require almost no downtime we would need to restart tugboat and remove current server Risks:

we would need to make sure the manager (server) is pointing to docker swarm that is older version of docker and storage driver as well as new one we would also need a good rolback strategy we would not move over production stuff right away (move test first and have a bake in period) Of Note:

No chance of tugboat dropping support if we don’t do this (they try to keep tugboat agnostic and not dependent on specific storage drivers) this upgrade to ZFS storage drivers could help tugboat with issues where we are running out of disk space.

gracekretschmer-metrostar commented 1 month ago

For Sprint 21, schedule a meeting to regroup and determine next steps with this work.