pnwstevan
commented
4 years ago Detailed documentation of upgrade procedure for future reference/onboarding
tl;dr
Detailed documentation of first production deploy; process itself and documentation is likely excessive, but consider this essentially upgrading Grafana the hard way as a bit of a forcing function to learn a bunch of things. Hopefully this is helpful for future reference and onboarding other Ops team members.
Thoughts/Notes:
- Seems like some opportunity to optimize the changes to seed job schedule; a bit of manual work to reconcile the intended/actual state.
- Felt a bit blind when pulling the trigger, other than waiting for Grafana to BRD and manually checking things out; likely some opportunity for improved visibility with logging.
- Should have potentially suppressed any alerts?
- Consider making use of deployment/annotation API in Grafana for historical record of upgrade (ie: change) in timeseries graphs.
Prerequisites:
Access and Auth:
- [ ] SSH keys and AWS Access as noted here
- [ ] GitHub Access
- [ ] Added to the VA Org
- [ ] Commit bit on the devops repo
- [ ] AWS Setup
- [ ] Keys and MFA
- [ ] DVSA AWS Console Access
- [ ] Everything in the internal tools doc setup and working
Local Tools for Dev & Testing:
- [ ] Git
- [ ] Vagrant
- [ ] VirtualBox
- [ ] Docker
- [ ] Ansible
- [ ] Python Stuffs (Pyenv, pip, Python 3.x, 2.x, etc..)
- [ ] AWS CLI
- [ ] aws-mfa tool (enables 2MFA for access to AWS resources via CLI)
- [ ] credstash
Upgrade Steps:
-
Ensure all preqs and access per above is setup - better to do this now than get blocked in the middle of a workflow.
-
Pre-work - gather context, requirements, etc..
- Reviewed the initial request.
- Let the team that owns the app know I'll be working on it.
- Reviewed the Grafana Releases and Release Notes.
- Note: I determined there had been a version bump since the request which looked like it made sense to upgrade to.
-
Making the changes for upgrade
- Clone the master branch of the respective repo locally (in this case, Grafana configs are in the devops repo)
- Create and checkout a new local branch
- Make the respective changes to enable upgrade:
- Version bump in Ansible config for Grafana
-
Test the upgrade locally
- Ensure you have a valid AWS token and working CLI
- Can test with
aws sts get-caller-identity
- Can test with
- Review and follow guide for using Vagrant
- Go to your local working directory with checked out branch then
~/devops/ansible - Setup Python
- Virtual Environment
pip install -r requirements.txt- Build with changes:
export APP=grafana-vagov;export ENV=vagov-dev; export REF=master; vagrant up --provision-with build - Forward port to connect locally:
vagrant ssh -- -L 3000:localhost:3000 - Login page @ http://localhost:3000 - yay!
- Boo - can't login cause no DB :(
- Reset admin password!
- From SSH shell into Vagrant Host:
- Find container:
docker ps- Shell into container:
docker exec -ti <container id> bash - Reset admin password:
grafana-cli admin reset-admin-password <new password
- Shell into container:
- Find container:
- Can log in to locally upgraded test build!
- Notice there are warnings about unsigned plugins, which I saw in release notes and need to make another change.
- Destroy Vagrant build:
vagrant destroy - Make change to allow unsigned plugins
- Rebuild again with above steps
- SUCCESS! :D
- Don't submit a PR yet, since it will get built, deployed and released automatically per Jenkins schedule (more on this below).
- Ensure you have a valid AWS token and working CLI
-
Prepare for Production Upgrade
- Since there is some risk, and my first deploy going to be extra cautious and do this manually.
- Disable daily build for Grafana by removing cron in seed job:
- Commit
- Push commit to master
- Submit Pull Request
- Also need to make this change on live production server:
- Login to Jhttp://jenkins.vfs.va.gov/
- Seed Job->Configure-DSL Script
- Make same change as above PR
- NOW we can submit our actual upgrade PR, without fear it will be deployed automatically.
- Commit and push to master
- Submit Pull Request for Upgrade
- Discuss with team in office hours; plan for upgrade 10am PST 8/25
- Announce ^ via Slack to original request thread, and in #vfs-engineers
-
Production Upgrade
- Get browser tabs open/ready:
- Jenkins Prod
- Grafana
- AWS Console:
- RDS (for Prod DB Backup / Monitoring)
- CloudWatch (created a simple dashboard for network/io from ELB serving Grafana)
- Build, Release & Deploy time!
- Backup Grafana Prod DB
- Find prod grafana DB in RDS
- Actions->Take Snapshot
- Confirm snapshot is complete
- Leave tab open for DB metrics in case needed for troubleshooting
- Build, Release, Deploy Grafana upgrade:
- Login to Jenkins
- BUILD!
- Build->grafana-vagov
- Build with Parameters
- Click on active Build #
- Monitor build with "Console Output"
- When complete, click link in "Scheduling Project:"
- RELEASE!
- Click on active Release #
- Monitor release with "Console Output"
- DEPLOY!
- Click on active Deploy #
- Monitor deploy with "Console Output"
- If successful, app should be up, which it was.
-
Testing / Validation
- Logged into Grafana
- Checked a bunch of charts
- Looked for anything obviously broken
- Monitored CW metrics and RDS charts
-
Post Deploy / Clean-up
- Notification upgrade was complete in previous Slack threads
- Re-enable seed job:
- Revert commit (could have submitted new commit/PR, thought this was cleaner)
- Re-enabled it on live Jenkins server (see above)
ricetj
Description
The Identity team and BE tools team would like to utilize some features in Grafana 7 (latest). This will be a major upgrade from 6 to 7 and it looks like there are going to be some issues with w/ backend plugins & signing.
Background/context/resources
Both Bennie and Keifer are willing to support this rollout.
Technical notes
ref: https://dsva.slack.com/archives/CJYRZK2HH/p1597089144037000?thread_ts=1597078047.032300&cid=CJYRZK2HH
Tasks
Definition of Done
Reminders