[devops] Add capability to rollback a code deploy

[franckc]

CC @mikeshultz

[mikeshultz]

This is an unintended side-effect of an intentional decison on the dshop pipeline to reduce other deployment issues. :-\

We have the ability to rollback, it's just not necessarily as simple as helm rollback experimental [number]. Since we aren't using helm to rollout image/build updates, it's not aware of these image changes. So a helm rollback would actually just alter the infra and still use the latest image.

Two ways I know this can be done now:

• Tag the previous (good) image as latest, run a helm deployment
• Patch the StatefulSet with the image you want again kubectl patch statefulset "experimental-dshop-backend-rinkeby" --type='json' -p='[{ "op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"gcr.io/origin-214503/experimental/dshop-backend:2020052719171590607078" }]'

I prefer the latter over the former, but want to do some other research before I make a recommendation. Just wanted to comment here in case of an emergency before we have something more friendly/permanent.

[mikeshultz]

Looks like kubectl rollout undo sts/experimental-dshop-backend-mainnet may be an option. Want to test it out.

Also, displaying revision history with kubectl is super uninformative.

$ kubectl rollout history sts/experimental-dshop-backend-rinkeby
statefulset.apps/experimental-dshop-backend-rinkeby 
REVISION
0
0
0
0
0
0
0
0
0
0
0
134
135
136
137
138
139
140
141
142
143
144

This may be due to --record not being used for the CI/CD patch:

      --record=false: Record current kubectl command in the resource annotation. If set to false, do not record the
command. If set to true, record the command. If not set, default to updating the existing annotation value only if one
already exists.

[mikeshultz]

This does appear to work(the dry run diff looks good):

kubectl rollout undo --dry-run=true sts/experimental-dshop-backend-rinkeby

Though --to-revision=[rev] is useless when all the recent revisions are 0. That's probably from not using --record but want to figure that out so we can target specific versions if we want.