search

openstax / napkin-notes

:notebook: virtual back-of-a-napkin notes (persistent non-structured discussions)
3 stars 8 forks source link

Deployment Process Postmortem #74

Open philschatz opened 7 years ago

philschatz commented 7 years ago

Here are some workflows describing the process.

Click here for rendered sequence diagrams

Current Process

The workflow described below occurred when releasing a hotfix to changing the MathJax CDN. I separated the developer sequence from the QA one to make it a little easier to grok.

Developer Sequence

This shows the steps a developer goes through to test and deploy to dev

image

Source... ```mermaid sequenceDiagram Developer ->> GitHub: Push a commit (sha) GitHub ->> Jenkins: Notify Push Event (sha) Jenkins -->> Jenkins: Build the tarball (sha) Jenkins ->> Package Server: Push the tarball (sha) Jenkins ->> GitHub: Notify Build Success (sha) Note Right of Developer: The following steps are for illustration only Developer ->> Rundeck: Request Deployment (sha) Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> Developer: Notify Deployment Success ```

QA Sequence

These steps describe what QA currently does to deploy and test once a Develoepr is donw. There are several manual error-prone steps and communication/coordination that need to happen in order to get it done.

Note: There are 16 manual steps in this workflow (in addition to the actual testing on QA/staging/production)

Note: Only one of these manual steps is because Jenkins listens to push events, not tag-creation events.

deploy-current svg

Source... ```mermaid sequenceDiagram QA ->> Developer: Request a tag Developer ->> GitHub: Create a tag (v2.0rc1) Note Right of Developer: Manually confirm that the commit matches the tag Developer ->> QA: Notify that tag is created QA ->> Ops: request deployment of the tag (v2.0rc1) Ops -->> Ops: Copy the dev tarball to the tagged tarball (v2.0rc1) Ops ->> Rundeck: Deploy tag to qa (v2.0rc1) Rundeck ->> Ops: Notify deployment completed Ops ->> QA: Notify deployment completed QA -->> QA: Test qa QA ->> Ops: Request deployment to staging (v2.0rc1) Ops ->> Rundeck: Deploy tag to staging (v2.0rc1) Rundeck ->> Ops: Notify deployment completed Ops ->> QA: Notify deployment completed QA -->> QA: Test staging QA -->> QA: Get authorization for deployment QA ->> Developer: Request that tag is created (v2.0) Developer ->> GitHub: Create the tag v2.0 Note left of Developer: hope that this tag is the same sha as v2.0rc1 Developer ->> QA: Notify that tag is created (v2.0) QA ->> Ops: Request deployment to staging (v2.0rc1) Ops ->> Rundeck: Deploy tag to staging (v2.0rc1) Rundeck ->> Ops: Notify deployment completed Ops ->> QA: Notify deployment completed QA -->> QA: Test staging QA ->> Ops: Request deployment to production (v2.0) Ops ->> Rundeck: Deploy tag to production (v2.0) Rundeck -->> Rundeck: Deploy Rundeck ->> Ops: Notify deployment completed Ops ->> QA: Notify deployment completed QA -->> QA: Smoketest Production ```

Proposal

In this, the communication between groups is minimized and the number of manual (error-prone) steps is also minimized.

for QA deployment...

Note: "Ops" and "Developer" are missing from this :smile:

deploy-ideal-qa svg

Source... ```mermaid sequenceDiagram QA ->> Rundeck: Request QA Deployment (sha) Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> QA: Notify deployment completed QA -->> QA: Test QA QA ->> Rundeck: Request Staging Deployment (sha) Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> QA: Notify deployment completed QA -->> QA: Test Staging QA ->> Rundeck: Request Production Deployment (sha) Note Right of Rundeck: Check that the sha is a descendant of production's sha Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> QA: Notify deployment completed QA -->> QA: Smoketest Production ```

Combined dev and QA deployment workflow

Note: There are only 6 manual steps in this combined Dev+QA workflow.

Note: Ops is missing from this and Developer & QA only need to communicate once :smile:

Note: Devs or QA or whoever can create and use tags on top but it's optional

deploy-ideal-all svg

Source... ```mermaid sequenceDiagram Developer ->> GitHub: Push a commit (sha) GitHub ->> Jenkins: Notify Push Event (sha) Jenkins -->> Jenkins: Build the tarball (sha) Jenkins ->> Package Server: Push the tarball (sha) Jenkins ->> GitHub: Notify Build Success (sha) Developer ->> Rundeck: Request Deployment (sha) Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> Developer: Notify Deployment Success Developer ->> QA: Notify that the sha is ready for testing (sha) Note right of Developer: Developer is done at this point QA ->> Rundeck: Request QA Deployment (sha) Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> QA: Notify deployment completed QA -->> QA: Test QA QA ->> Rundeck: Request Staging Deployment (sha) Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> QA: Notify deployment completed QA -->> QA: Test Staging QA ->> Rundeck: Request Production Deployment (sha) Note Right of Rundeck: Check that the sha is a descendant of production's sha Rundeck -->> Rundeck: Deploy (sha) Rundeck ->> QA: Notify deployment completed QA -->> QA: Smoketest Production ```

And that's it! Whaddya think?