chore: upgrade to Tugboat v2.21.12 - v2.22.7

q0rban commented 3 years ago

The latest Tugboat release upgrades the version of Mongo. CHANGELOG is described on Slack.

It's recommended to do this release during off-hours to avoid disruption, and since the mongo database is affected, a backup prior to the deployment is advised.

AC:

[ ] Tugboat is updated to v2.22.7

Tasks for v2.21.11 to v2.21.12 (see comments for tasks for later versions):

[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Backup the Mongo database: util/backup.sh
[x] Stop Tugboat: tbctl stop
[x] Fetch the latest tags: git fetch --tags
[x] Checkout this version: git checkout v2.21.12
[x] Build: make && make install
[x] Start Tugboat again: tbctl start
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status OR curl --insecure https://tugboat.vfs.va.gov/_status
[x] Verify the new version of mongo is 4.0: docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"

q0rban commented 3 years ago

v2.21.13 includes a step to increase the Mongo featureCompatibilityVersion to 4.0 using the setFeatureCompatibilityVersion command. According to the MongoDB upgrade instructions, it is recommended to allow a burn-in period before doing this. We waited one week, but you may wait lower amounts of time.

Tasks:

[x] Wait some time after v2.21.12 has been deployed
[x] Verify that the current running version of Mongo is 4.0 docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Fetch the latest tags: git fetch --tags
[x] Checkout v2.21.13: git checkout v2.21.13
[x] Build: make && make install
[x] Reload Tugboat: tbctl reload
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify that the featureCompatibilityVersion of the Mongo service is 4.0 (Run docker exec -it tugboat-mongo[TAB] mongo and then db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

q0rban commented 3 years ago

v2.21.14 upgrades the MongoDB service version to 4.2. This is a similar release to v2.21.12: we upgrade the version of the service that is running, but the featureCompatibilityVersion will stay at the 4.0 that was set with v2.21.13. To deploy v2.21.14:

Tasks:

[x] Wait some time after v2.21.13 has been deployed.
[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Backup the Mongo database: util/backup.sh
[x] Stop Tugboat: tbctl stop
[x] Fetch the latest tags: git fetch --tags
[x] Checkout this version: git checkout v2.21.14
[x] Build: make && make install
[x] Start Tugboat again: tbctl start
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify the new version of mongo is 4.2: docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"

q0rban commented 3 years ago

v2.21.15 includes a step to increase the Mongo featureCompatibilityVersion to 4.2 using the setFeatureCompatibilityVersion command. According to the MongoDB upgrade instructions, it is recommended to allow a burn-in period before doing this. We waited one week, but you may wait lower amounts of time.

Tasks:

[x] Wait some time after v2.21.14 has been deployed
[x] Verify that the current running version of Mongo is 4.2 docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Fetch the latest tags: git fetch --tags
[x] Checkout v2.21.15: git checkout v2.21.15
[x] Build: make && make install
[x] Reload Tugboat: tbctl reload
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify that the featureCompatibilityVersion of the Mongo service is 4.2 (Run docker exec -it tugboat-mongo[TAB] mongo and then db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

q0rban commented 3 years ago

EDIT: v2.21.16 contained a UI bug that hid the display of base previews. While v2.21.16 is the actual release that contains the MongoDB upgrade, I've edited these notes to use v2.21.17 instead of v2.21.16 so that you do not experience that bug.

v2.21.17 upgrades the MongoDB service version to 4.4. This is a similar release to v2.21.12: we upgrade the version of the service that is running, but the featureCompatibilityVersion will stay at the 4.2 that was set with v2.21.15. To deploy v2.21.17:

Tasks:

[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Backup the Mongo database: util/backup.sh
[x] Stop Tugboat: tbctl stop
[x] Fetch the latest tags: git fetch --tags
[x] Checkout this version: git checkout v2.21.17
[x] Build: make && make install
[x] Start Tugboat again: tbctl start
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify the new version of mongo is 4.4: docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"

ElijahLynn commented 2 years ago

Thanks for all these update comments @q0rban, super helpful!

q0rban commented 2 years ago

v2.21.20 includes a step to increase the Mongo featureCompatibilityVersion to 4.4 using the setFeatureCompatibilityVersion command. According to the MongoDB upgrade instructions, it is recommended to allow a burn-in period before doing this. We waited one week, but you may wait lower amounts of time.

Tasks:

[x] Wait some time after v2.21.17 has been deployed
[x] Verify that the current running version of Mongo is 4.4 docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Fetch the latest tags: git fetch --tags
[x] Checkout v2.21.20: git checkout v2.21.20
[x] Build: make && make install
[x] Reload Tugboat: tbctl reload
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify that the featureCompatibilityVersion of the Mongo service is 4.4 (Run docker exec -it tugboat-mongo[TAB] mongo and then db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

cmaeng commented 2 years ago

@ndouglas is recommending to perform each update individually to catch up to the current version which may affect timing beyond one sprint. Testing will also be tricky as our current infrastructure isn't set up to test well. His other concern is around the demo environments and having working data in the system.

We'll try to pick this all apart during sprint planning to get a good sense of what this all will entail.

ndouglas commented 2 years ago

Note this comment too:

v2.21.13 includes a step to increase the Mongo featureCompatibilityVersion to 4.0 using the setFeatureCompatibilityVersion command. According to the MongoDB upgrade instructions, it is recommended to allow a burn-in period before doing this. We waited one week, but you may wait lower amounts of time.

olivereri commented 2 years ago

Current version:

[root@tugboat]# tugboat version
Client: 2.21.11
Server: 2.21.11

q0rban commented 2 years ago

Glad to see this one getting dusted off! We've got two more steps to add to this, and then you all will be smooth sailing for future upgrades.

v2.22.1 upgrades the MongoDB service version to 5.0. This is a similar release to v2.21.12: we upgrade the version of the service that is running, but the featureCompatibilityVersion will stay at the 4.4 that was set with v2.21.20. To deploy v2.22.1:

Tasks:

[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Backup the Mongo database: util/backup.sh
[x] Stop Tugboat: tbctl stop
[x] Fetch the latest tags: git fetch --tags
[x] Checkout this version: git checkout v2.22.1
[x] Build: make && make install
[x] Start Tugboat again: tbctl start
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify the new version of mongo is 5.0: docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"

q0rban commented 2 years ago

v2.22.3 includes a step to increase the Mongo featureCompatibilityVersion to 5.0 using the setFeatureCompatibilityVersion command. According to the MongoDB upgrade instructions, it is recommended to allow a burn-in period before doing this. We waited one week, but you may wait lower amounts of time.

Tasks:

[x] Wait some time after v2.22.1 has been deployed
[x] Verify that the current running version of Mongo is 5.0 docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Fetch the latest tags: git fetch --tags
[x] Checkout v2.22.3: git checkout v2.22.3
[x] Build: make && make install
[x] Reload Tugboat: tbctl reload
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status
[x] Verify that the featureCompatibilityVersion of the Mongo service is 5.0 (Run docker exec -it tugboat-mongo[TAB] mongo and then db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

q0rban commented 2 years ago

Thanks for your patience in this process, once you're on mongo 5.0, we won't need to do this again, as 5.0 allows for backwards compatibility. Once you all are on version v2.22.3, you'll be free to upgrade to newer versions (such as v2.22.7) without having to carefully step through in this manner.

[x] Change directories to the Tugboat root: cd /opt/tugboat
[x] Fetch the latest tags: git fetch --tags
[x] Checkout the tag (e.g. git checkout v2.22.7)
[x] Build: make && make install
[x] Reload Tugboat: tbctl reload
[x] Wait for Tugboat to start up successfully. For the lazy: wget --quiet -O- --tries=100 --retry-on-http-error=404,502,503 --retry-connrefused https://[tugboat-dashboard-host]/_status

Here's the changelog from v2.21.12 to v2.22.7 (which is the latest release as of today Sep 20, 2022):

# Change Log

## v2.22.7 - 23 August, 2022

* #4323 chore: bump socket.io from 4.4.1 to 4.5.1
* #1850 feat: add browser notifications when previews are ready
* #4356 feat: include the domain as the namespace for traces
* #3352 feat: allow for reauthenticating repository from the UI
* #4357 feat: add service specific telemetry options
* #4361 fix: correct breadcrumb typo of Downloads
* #4338 chore: bump terser from 5.9.0 to 5.14.2
* #4352 chore: bump xterm from 4.14.1 to 4.19.0
* #4282 chore: bump react-router-dom from 5.3.0 to 5.3.3
* #4363 fix: exit agent process if agent-proxy throws uncaught exception
* #4373 fix: establish socket with agent-proxy over an interval
* #4381 fix: fix typo in cli list help
* #4367 chore: bump redoc from 2.0.0-rc.56 to 2.0.0-rc.74
* #4372 chore: bump prettier from 2.4.1 to 2.7.1
* #4320 chore: bump parse-url from 6.0.0 to 6.0.2
* #4376 fix: tar up older backups to reduce disk usage

## v2.22.6 - 23 July, 2022

* #4301 chore: bump glob from 7.2.0 to 7.2.3
* #4300 chore: bump sass-loader from 12.3.0 to 12.6.0
* #4297 chore: bump loglevel from 1.7.1 to 1.8.0
* #4311 chore: bump winston from 2.4.5 to 2.4.6
* #4299 chore: bump @opentelemetry/sdk-trace-node from 1.2.0 to 1.3.1
* #4312 chore: bump css-minimizer-webpack-plugin from 3.1.1 to 3.4.1
* #4316 chore: bump query-string from 7.0.1 to 7.1.1
* #4315 chore: bump eslint-plugin-jsx-a11y from 6.4.1 to 6.6.0
* #4321 fix: restore chrome dependencies in node:14 image
* #4303 fix: remove e2e hard coded classes
* #4255 feat: local dev support for arm64 using parallels
* #4307 chore: bump jpeg-js from 0.4.3 to 0.4.4
* #4331 fix: choose appropriate agent during tugboat rebuild screenshots
* #3683 fix: handle splash screen for suspending previews
* #4326 fix: handle GitLab refresh tokens on the server
* #3429 feat: add option to not build / rebuild draft PRs
* #4330 chore: bump moment from 2.29.3 to 2.29.4
* #4329 chore: bump express-session from 1.17.2 to 1.17.3
* #4324 fix: handle gitlab refresh token race condition
* #4324 fix: ensure refreshed token is used after first gitlab api call

## v2.22.5 - 21 June, 2022

* #4249 chore: bump moment from 2.29.1 to 2.29.3
* #4220 chore: bump prismjs from 1.25.0 to 1.27.0
* #4204 chore: security upgrade sharp from 0.29.2 to 0.29.3
* #4215 chore: Bump simple-get from 3.1.0 to 3.1.1
* #4046 chore: Bump path-parse from 1.0.6 to 1.0.7
* #4099 chore: Bump tmpl from 1.0.4 to 1.0.5
* #4251 chore: Bump core-js from 3.17.3 to 3.22.2
* #4216 chore: Bump follow-redirects from 1.13.0 to 1.14.8
* #4150 chore: Bump passport-oauth2 from 1.5.0 to 1.6.1
* #4180 chore: Upgrade parse-link-header from 1.0.1 to 2.0.0
* #4211 chore: Bump @babel/plugin-transform-runtime from 7.14.5 to 7.17.0
* #4259 chore: Bump node-schedule from 2.0.0 to 2.1.0
* #4260 chore: Bump semver from 7.3.5 to 7.3.7
* #4264 chore: bump @babel/preset-env from 7.15.6 to 7.17.10
* #4263 chore: Bump mocha from 9.1.3 to 9.2.2
* #4258 chore: Bump chart.js from 3.6.0 to 3.7.1
* #4190 chore: Security upgrade socket.io from 4.3.1 to 4.4.0
* #4268 chore: Bump eslint-plugin-import from 2.24.2 to 2.26.0
* #4285 chore: Security upgrade sharp from 0.29.3 to 0.30.5
* #4283 chore: bump mini-css-extract-plugin from 2.4.3 to 2.6.0
* #3966 feat: Add quick search to repos, projects and previews
* #4233 feat: provide open telemetry tracing to Tugboat services
* #4296 chore: bump eslint-plugin-react-hooks from 4.2.0 to 4.6.0
* #4295 chore: bump protobufjs from 6.11.2 to 6.11.3
* #4291 chore: bump postcss-scss from 4.0.2 to 4.0.4
* #4270 chore: bump express from 4.17.1 to 4.18.1
* #4269 chore: bump nodemailer from 6.7.0 to 6.7.5
* #4298 chore: bump chart.js from 3.7.1 to 3.8.0
* #4281 fix: refresh oauth gitlab tokens for repos
* #4302 fix: allow searchInfo to be empty

## v2.22.4 - 16 May, 2022

* #4252 fix: clean orphaned containers via iteration
* #4277 fix: switch tugboat.qa to tugboatqa.com

## v2.22.3 - 16 April, 2022

* #4222 fix: write local reference to tag when fetching
* #4226 fix: use bitbucket workspace.slug instead of owner.nickname / owner.username
* #4231 fix: write error to http-proxy response only if it is present
* #4229 fix: increase apache ServerLimit to accommodate more traffic (ServerLimit & MaxRequestWorkers from 1024 to 3072)
* #3863 feat: upgrade MongoDB featureCompatibilityVersion to 5.0
* #4238 fix: clean up orphaned preview data
* #4246 fix: do not allow clean-agent.sh to run on manager servers
* #4234 feat: change splash page http response from 503 to 418

## v2.22.2 - 12 February, 2022

* #4201 fix: update gitlab oauth scope to api
* #4206 fix: only trigger GitLab MR webhook when new commits are pushed
* #4036 fix: ensure correct agent when base preview is rebuilt
* #4212 fix: emit error 1132 if pulling docker image fails

## v2.22.1 - 22 January, 2022

* #4173 tests: update splash template snapshot
* #4157 fix: catch and cleanup temp git previews in git.getYML
* #3863 chore: update mongo images to 5.0
* #4181 fix: remove errant exit command from git.getYML
* #4136 chore: clean up preview sort regression technical debt
* #4193 fix: increase email attachment size to 10mb
* #4196 fix: log smtp errors appropriately

## v2.21.20 - 23 December, 2021

* #4170 fix: use websocket transport for splash page

## v2.21.19 - 13 December, 2021

* #4168 fix: prevent hamburger click from bubbling immediately

## v2.21.18 - 11 December, 2021

* #4093 chore: update npm deps to latest majors
* #4093 fix: set `strictQuery: false` on all schemas, passport `pid` querying
* #4129 docs:  document the current Enzyme testing decision
* #4159 feat: during agent cleanup, only delete dangling git previews for the current agent
* #4104 chore: React17 update, use community enzyme adapter
* #4122 chore: set mongo featureCompatibilityVersion to 4.4
* #4162 fix: correct grammar in connected provider message
* #4132 feat: add specific error messages and more validation to Envvars Import
* #4166 fix: use --no-recurse-submodules during git fetch
* #4122 fix: continue using batchSize in mongorestore with Mongo 4.4
* #4157 fix: use alpine:3.14 for compatibility with older git SSH implementations

## v2.21.17 - 02 November, 2021

* #4134 fix: list base previews without sorting

## v2.21.16 - 01 November, 2021

* #4122 chore: update mongo images to 4.4
* #3898 feat: remember Preview sorting
* #4127 fix: Make sure ui server exits if error during startup

## v2.21.15 - 09 October, 2021

* #4086 Set mongo featureCompat to 4.2
* #4108 Use --batchSize=10 in mongorestore
* #4117 fix: accept expires param at v3 api preview patch endpoint

## v2.21.14 - 25 September, 2021

* #4086 Upgrade mongodb images to 4.2
* #3681 CLI help fixes
* #4092 Update dependencies to next minors
* #4094 Document agent mongo upgrades
* #4037 Fix build/runvars cli update crash

## v2.21.13 - 11 September, 2021

* #4059 Upgrade mongoDB featCompat flag to 4.0
* #4065 release an upgrade lock if a server or ui upgrade fails
* #3923 allow renaming of environment variables
* #4057 document how upgrades are performed on server and ui services
* #4073 Restore copy / paste ability in xterm for Chrome users
* #4038 Enable prefers-reduced-motion for visualdiffs
* #4071 Add form validation for custom environment variables

## v2.21.12 - 28 August, 2021

* #3863 Upgrade Mongo services to Mongo 4.0
* #4035 Fix bug on New Repository form that scrolled user to empty page
* #4063 Prevent CLI from erroring when listing API keys with verbose option
* #4058 Implement Archictecture Decision Records using log4brains
* #4047 Update the logo on the API
* #4066 Add support for screen readers for job logs, service output, and TuSH

q0rban commented 2 years ago

Please reach out to support@tugboatqa.com if you'd like to schedule some time to work on this together with a Tugboat engineer.

cmaeng commented 2 years ago

converting this to an epic so that each version upgrade can be its own issue (approximately 9-10ish versions). each upgrade may be between 3-8 points each, with things getting more efficient as we go on.

ElijahLynn commented 2 years ago

Starting v2.21.11 > v2.22.12 upgrade now:

git status 
HEAD detached at v2.21.11                  

docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"   
mongo:3

util/backup.sh

output

# util/backup.sh 2022-09-26T23:53:43.478+0000 writing admin.system.version to 2022-09-26T23:53:43.478+0000 done dumping admin.system.version (1 document) 2022-09-26T23:53:43.478+0000 writing tugboat.logs to 2022-09-26T23:53:43.478+0000 writing tugboat.jobs to 2022-09-26T23:53:43.479+0000 writing tugboat.statistics to 2022-09-26T23:53:43.479+0000 writing tugboat.trends to 2022-09-26T23:53:43.805+0000 done dumping tugboat.trends (130146 documents) 2022-09-26T23:53:43.805+0000 writing tugboat-ui.sessions to 2022-09-26T23:53:43.841+0000 done dumping tugboat-ui.sessions (19480 documents) 2022-09-26T23:53:43.841+0000 writing tugboat.networks to 2022-09-26T23:53:43.847+0000 done dumping tugboat.networks (3584 documents) 2022-09-26T23:53:43.847+0000 writing tugboat.services to 2022-09-26T23:53:43.849+0000 done dumping tugboat.services (153 documents) 2022-09-26T23:53:43.850+0000 writing tugboat.keys to 2022-09-26T23:53:43.850+0000 done dumping tugboat.keys (94 documents) 2022-09-26T23:53:43.850+0000 writing tugboat-ui.passports to 2022-09-26T23:53:43.851+0000 done dumping tugboat-ui.passports (82 documents) 2022-09-26T23:53:43.851+0000 writing tugboat-ui.users to 2022-09-26T23:53:43.853+0000 done dumping tugboat-ui.users (82 documents) 2022-09-26T23:53:43.853+0000 writing tugboat.previews to 2022-09-26T23:53:43.856+0000 done dumping tugboat.previews (59 documents) 2022-09-26T23:53:43.856+0000 writing tugboat.repos to 2022-09-26T23:53:43.857+0000 done dumping tugboat.repos (5 documents) 2022-09-26T23:53:43.857+0000 writing tugboat.registries to 2022-09-26T23:53:43.857+0000 done dumping tugboat.registries (4 documents) 2022-09-26T23:53:43.857+0000 writing tugboat-ui.permissions to 2022-09-26T23:53:43.858+0000 done dumping tugboat-ui.permissions (4 documents) 2022-09-26T23:53:43.858+0000 writing tugboat-ui.tiers to 2022-09-26T23:53:43.858+0000 done dumping tugboat-ui.tiers (3 documents) 2022-09-26T23:53:43.858+0000 writing tugboat.webhooks to 2022-09-26T23:53:43.858+0000 done dumping tugboat.webhooks (3 documents) 2022-09-26T23:53:43.858+0000 writing tugboat.upgrades to 2022-09-26T23:53:43.859+0000 done dumping tugboat.upgrades (3 documents) 2022-09-26T23:53:43.859+0000 writing tugboat.projects to 2022-09-26T23:53:43.859+0000 done dumping tugboat.projects (3 documents) 2022-09-26T23:53:43.859+0000 writing tugboat-ui.upgrades to 2022-09-26T23:53:43.860+0000 done dumping tugboat-ui.upgrades (1 document) 2022-09-26T23:53:43.860+0000 writing tugboat.agents to 2022-09-26T23:53:43.860+0000 done dumping tugboat.agents (1 document) 2022-09-26T23:53:43.860+0000 writing tugboat.messages to 2022-09-26T23:53:43.860+0000 done dumping tugboat.messages (0 documents) 2022-09-26T23:53:43.860+0000 writing tugboat-ui.invites to 2022-09-26T23:53:43.861+0000 done dumping tugboat-ui.invites (0 documents) 2022-09-26T23:53:43.861+0000 writing tugboat-ui.locks to 2022-09-26T23:53:43.861+0000 done dumping tugboat-ui.locks (0 documents) 2022-09-26T23:53:43.861+0000 writing tugboat.visualdiffs to 2022-09-26T23:53:43.861+0000 done dumping tugboat.visualdiffs (0 documents) 2022-09-26T23:53:43.861+0000 writing tugboat.lighthouses to 2022-09-26T23:53:43.862+0000 done dumping tugboat.lighthouses (0 documents) 2022-09-26T23:53:43.862+0000 writing tugboat.locks to 2022-09-26T23:53:43.862+0000 done dumping tugboat.locks (0 documents) 2022-09-26T23:53:43.862+0000 writing tugboat.screenshots to 2022-09-26T23:53:43.863+0000 done dumping tugboat.screenshots (0 documents) 2022-09-26T23:53:44.399+0000 done dumping tugboat.statistics (615707 documents) 2022-09-26T23:53:46.447+0000 [####....................] tugboat.logs 1763581/9688165 (18.2%) 2022-09-26T23:53:46.447+0000 [#######.................] tugboat.jobs 924717/2999815 (30.8%) 2022-09-26T23:53:46.447+0000 2022-09-26T23:53:49.447+0000 [#########...............] tugboat.logs 3657841/9688165 (37.8%) 2022-09-26T23:53:49.447+0000 [#############...........] tugboat.jobs 1660753/2999815 (55.4%) 2022-09-26T23:53:49.447+0000 2022-09-26T23:53:52.447+0000 [#############...........] tugboat.logs 5477974/9688165 (56.5%) 2022-09-26T23:53:52.447+0000 [#################.......] tugboat.jobs 2206902/2999815 (73.6%) 2022-09-26T23:53:52.447+0000 2022-09-26T23:53:55.447+0000 [##################......] tugboat.logs 7292624/9688165 (75.3%) 2022-09-26T23:53:55.447+0000 [#####################...] tugboat.jobs 2735572/2999815 (91.2%) 2022-09-26T23:53:55.447+0000 2022-09-26T23:53:56.979+0000 [########################] tugboat.jobs 2999815/2999815 (100.0%) 2022-09-26T23:53:56.979+0000 done dumping tugboat.jobs (2999815 documents) 2022-09-26T23:53:58.447+0000 [######################..] tugboat.logs 9231839/9688165 (95.3%) 2022-09-26T23:53:59.392+0000 [########################] tugboat.logs 9688165/9688165 (100.0%) 2022-09-26T23:53:59.392+0000 done dumping tugboat.logs (9688165 documents)

tbctl stop

Stopping Tugboat...

  Service: agent                                                        [  OK  ]
  Service: apache                                                       [  OK  ]
  Service: api                                                          [  OK  ]
  Service: logs                                                         [  OK  ]
  Service: mail                                                         [  OK  ]
  Service: mongo                                                        [  OK  ]
  Service: proxy                                                        [  OK  ]
  Service: puppeteer                                                    [  OK  ]
  Service: redis                                                        [  OK  ]
  Service: registry                                                     [  OK  ]
  Service: scheduler                                                    [  OK  ]
  Service: server                                                       [  OK  ]
  Service: traefik                                                      [  OK  ]
  Service: ui                                                           [  OK  ]
  Service: webhooks                                                     [  OK  ]

git checkout v2.21.12
Previous HEAD position was 24e5da7b5 v2.21.11
HEAD is now at 715c51129 v2.21.12

make && make install

...
Done in 52.49s.
ln -sf "/opt/tugboat/bin/linux/tugboat" /usr/local/bin/tugboat
ln -sf "/opt/tugboat/bin/tbctl" /usr/local/sbin/tbctl

tbctl start                                       

Starting Tugboat...                                                               

  Checking Docker                                                       [  OK  ]  
  Network: tugboat                                                      [  OK  ]  
  Service: agent                                                        [  OK  ]  
  Service: apache                                                       [  OK  ]  
  Service: api                                                          [  OK  ]  
  Service: logs                                                         [  OK  ]  
  Service: mail                                                         [  OK  ]  
  Service: mongo                                                        [  OK  ]  
  Service: proxy                                                        [  OK  ]  
  Service: puppeteer                                                    [  OK  ]  
  Service: redis                                                        [  OK  ]  
  Service: registry                                                     [  OK  ]  
  Service: scheduler                                                    [  OK  ]  
  Service: server                                                       [  OK  ]  
  Service: traefik                                                      [  OK  ]  
  Service: ui                                                           [  OK  ]  
  Service: webhooks                                                     [  OK  ]

docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
mongo:4.0

We are now at v2.21.12. I tested some existing preview environments and they are coming up as expected. I am going to wait 1 day to go to v2.21.13 (to set the featureCompatibilityVersion) per Mongo guidelines to let a bake in period for the Mongo 3 to 4 upgrade. This is so that if we need to roll back to Mongo 3 for any data corruption issues it will be much less of a headache.

ElijahLynn commented 2 years ago

Update from v2.21.12 to v2.21.13 will be today at 3:30pm PT/6:30pm ET.

Posted in Slack to #sitewide-program, and x-posted to #sitewide-cms and #platform-cms.

ElijahLynn commented 2 years ago

Update from v2.21.12 to v2.21.13 is complete per instructions in comment above. Total downtime was 3:30:00-3:30:37.

[root@ip-172-31-1-159 tugboat]# git checkout v2.21.13
Previous HEAD position was 715c51129 v2.21.12
HEAD is now at 0f174d2c5 v2.21.13

make && make install

Done in 55.84s.
ln -sf "/opt/tugboat/bin/linux/tugboat" /usr/local/bin/tugboat
ln -sf "/opt/tugboat/bin/tbctl" /usr/local/sbin/tbctl

This is the downtime part:

[root@ip-172-31-1-159 tugboat]# tbctl reload                                      

Stopping Tugboat...                                                               

  Service: agent                                                        [  OK  ]  
  Service: api                                                          [  OK  ]  
  Service: logs                                                         [  OK  ]  
  Service: mail                                                         [  OK  ]  
  Service: proxy                                                        [  OK  ]  
  Service: puppeteer                                                    [  OK  ]  
  Service: registry                                                     [  OK  ]  
  Service: scheduler                                                    [  OK  ]  
  Service: server                                                       [  OK  ]  
  Service: ui                                                           [  OK  ]  
  Service: webhooks                                                     [  OK  ]  

Starting Tugboat...

  Checking Docker                                                       [  OK  ]
  Network: tugboat                                                      [  OK  ]
  Service: agent                                                        [  OK  ]
  Service: api                                                          [  OK  ]
  Service: logs                                                         [  OK  ]
  Service: mail                                                         [  OK  ]
  Service: proxy                                                        [  OK  ]
  Service: puppeteer                                                    [  OK  ]
  Service: registry                                                     [  OK  ]
  Service: scheduler                                                    [  OK  ]
  Service: server                                                       [  OK  ]
  Service: ui                                                           [  OK  ]
  Service: webhooks                                                     [  OK  ]

> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
{ "featureCompatibilityVersion" : { "version" : "4.0" }, "ok" : 1 }

I verified that one of the previews is still coming up as expected.

Tomorrow we will go from v2.21.13 to v2.21.14.

ElijahLynn commented 2 years ago

Today's update from v2.21.13 to v2.21.14 (see https://github.com/department-of-veterans-affairs/va.gov-cms/issues/6306#issuecomment-929231281) will be at 4pm PT/7pm ET. Notified #sitewide-program and x-posted in #sitewide-cms and #cms-platform too.

This will complete https://github.com/department-of-veterans-affairs/va.gov-cms/issues/10883.

ElijahLynn commented 2 years ago

V2.21.14 upgrade is complete :heavy_check_mark:. No issues noted.

make && make install

...
Done in 55.76s.
ln -sf "/opt/tugboat/bin/linux/tugboat" /usr/local/bin/tugboat
ln -sf "/opt/tugboat/bin/tbctl" /usr/local/sbin/tbctl

[root@ip-172-31-1-159 tugboat]# docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
mongo:4.2

V2.21.15 && v2.21.17 will be performed tomorrow at 5pm.

ElijahLynn commented 2 years ago

v2.21.15 and v2.21.17 have been deployed.

Downtime today in PT v2.21.15 - 5:55:20-5:55:55 (35s) v2.21.17 - 6:00:05-6:05:50 (5m45s)

[root@ip-172-31-1-159 tugboat]# git status
HEAD detached at v2.21.17

[root@ip-172-31-1-159 tugboat]# docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
mongo:4.4

ElijahLynn commented 2 years ago

v2.21.17 to v.21.20 complete:

Downtime 4:00:05-4:00:50pm PT (45s).

> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1} )
{ "featureCompatibilityVersion" : { "version" : "4.2" }, "ok" : 1 }

[root@ip-172-31-1-159 tugboat]# git checkout v2.21.20
Previous HEAD position was 40df2182d Merge pull request #4135 from Lullabot/release/v2.21.17
HEAD is now at 0291a72a2 v2.21.20

make && make install
...
success

> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1} )
{ "featureCompatibilityVersion" : { "version" : "4.4" }, "ok" : 1 }

ElijahLynn commented 2 years ago

v2.21.20 to v2.22.1 upgrade complete, total downtime 8m. We are now on Mongo 5.0.

[root@ip-172-31-1-159 tugboat]# git checkout v2.22.1
Previous HEAD position was 0291a72a2 v2.21.20
HEAD is now at 216e28f35 v2.22.1

make && make install
...
successful

[root@ip-172-31-1-159 tugboat]# docker ps --format "{{.Image}}" --filter "name=tugboat-mongo*"
mongo:5.0

ElijahLynn commented 2 years ago

Two more to go! v2.22.3 && v2.22.7, both of which will be done on Monday at 4pm PT!

q0rban commented 2 years ago

Now that you are on Mongo 5.0 containers, you can actually go straight to v2.22.7!

ElijahLynn commented 2 years ago

https://github.com/department-of-veterans-affairs/va.gov-cms/issues/10958 is done :heavy_check_mark:.

[root@ip-172-31-1-159 tugboat]# git checkout v2.22.3
Previous HEAD position was 216e28f35 v2.22.1
HEAD is now at 233b3432e v2.22.3

make && make install
...
successful

tbctl reload
...
successful

> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1} )
{ "featureCompatibilityVersion" : { "version" : "5.0" }, "ok" : 1 }

ElijahLynn commented 2 years ago

Now that you are on Mongo 5.0 containers, you can actually go straight to v2.22.7!

I didn't see this in time, and went to v2.22.3 first.

ElijahLynn commented 2 years ago

https://github.com/department-of-veterans-affairs/va.gov-cms/issues/10960 is done :heavy_check_mark:.

[root@ip-172-31-1-159 tugboat]# git checkout v2.22.7
Previous HEAD position was 233b3432e v2.22.3
HEAD is now at e5fc9d629 v2.22.7

make && make install
...
success

tbctl reload

ElijahLynn commented 2 years ago

All Tugboat updates have been applied, closing this issue/epic! :rocket:

department-of-veterans-affairs / va.gov-cms