Create new version and go process

Spike Description

The goal of the spike is to perform and document operations that support use cases, listed below, which target three personas: DevOps Engineer, App Team Developer, and App Team Release Manager. In general, the goal is to use Cobalt without having to use the 'fork & go' method of adoption, but rather create and support an AZ DO Pipeline configuration that points directly to the App Team repo (vs 'Importing' the microsoft/cobalt repo, or forks of that repo).

As a user, I'd like to source a template to a version and use this in my code for deployments, in order to have a simple deployment process

Acceptance Criteria

Reference: [Done-Done Checklist] (https://github.com/Microsoft/code-with-engineering-playbook/blob/master/Engineering/BestPractices/DoneDone.md)

[ ] Should use version for source of template
[ ] Should have a simple way to deploy using existing CI/CD
[ ] Should have configurable variables provided to app dev team

Support the following use cases:

[ ] DevOps-Eng: onboard an app-team into a Cobalt new AZDO pipeline project
- DevOps-Eng sends App-Dev the infra folder
- App-Dev puts infra at the root of their git repo, adjusts values therein, pushes change to master. Then notifies DevOps.
- DevOps-Eng creates new AZDO Project, points to infra, does a devint deployment, perhaps qa and/or release envs, too.
- DevOps-Eng notifies App-Team to use the new pipeline for CI/CD, releases, etc.
[ ] App-Dev commits & PRs code, merges to master, wants new WIP/RC build to go live on devint or perhaps qa, etc.
[ ] App-Dev uses the AZ DO pipeline to trigger a deployment to the desired env
[ ] App-Release-Manager wants to release a new version of App to production
- AppReleaseManager uses the AS DO Pipeline to trigger a production deployment
[ ] DevOps Guy: wants to get App-Team to use a new version of Cobalt and/or it's templates
- DevOp-Eng, or App-Dev, alters the infra files to point to a new tagged release of either the Cobalt bits (either at microsoft/cobalt@vNext//infra/templates/az-service-single-region or better, at foo.org@v6//cobalt-template-az-service-single-region or maybe foo.org@v6//co-template-std-microservice) *

Also, here are a few points that need to be addressed:

Keep the scope of work tight (it is a spike). Throw-away code. Keep docs (place in ``).
Prepare user stories for work in the following sprint.
Container-ization should be secondary, as well as using secrets in the demo app, etc.

Resources

[ ] Docs will be within https://github.com/microsoft/cobalt/tree/master/design-reference

Tasks

[ ] Create sequence diagrams for the various use cases
[ ] Config & run AZ DO Pipeline for the echo api app (sanity check)
[ ] Alter Cobalt and pipeline config to use yaml and tf from the echo api sub-dirs (and still work)
[ ] Create a repo housing 'only' the echo api app, plus the yaml and tf, mentioned above, and get Cobalt working against this new repo (so changes to the api app repo wind up triggering an AZ DO Cobalt Pipeline.

Notes from Aug 27 meeting with @iphilpot

We met to bounce ideas and set scope.

[ ] @ThePhen: edit the opening comment of this issue to reflect the following.
[ ] @ThePhen: create a throw-away branch for the spike.

Use Cases to spike

DevOps Guy: onboard an app-team into a Cobalt new AZDO pipeline project
- DevOps sends AppDev the infra folder
- AppDev puts infra at the root of their git repo, adjusts values therein, pushes change to master. Then notifies DevOps.
- DevOps creates new AZDO Project, points to infra, does a devint deployment, perhaps qa and/or release envs, too.
- DevOps notifies AppTeam to use the new pipeline for CI/CD, releases, etc.
AppDev commits & PRs code, merges to master, wants new WIP/RC build to go live on devint or perhaps qa, etc.
- AppDev uses the AZ DO pipeline to trigger a deployment to the desires env
AppReleaseManager wants to release a new version of App to production
- AppReleaseManager uses the AS DO Pipeline to trigger a production deployment
DevOps Guy: wants to get AppTeam to use a new version of Cobalt and/or it's templates
- DevOps, or AppDev, alters the infra files to point to a new tagged release of either the Cobalt bits (either at microsoft/cobalt@vNext//infra/templates/az-service-single-region or better, at foo.org@v6//cobalt-template-az-service-single-region or maybe foo.org@v6//co-template-std-microservice) *
Keep the scope of work tight (it is a spike). Throw-away code. Keep docs (place in ``). Prepare user stories for work in the following sprint.
Container-ization should be secondary, as well as using secrets in the demo app, etc.
external ref link should follow this reference format: github.com/microsoft/bedrock?ref=0.11.0//cluster/azure/vnet

Example Artifacts

Example of main.tf (this is rough/pseudo-code)

module "az-isolated-service-single-region" {
 source = "../az-isolated-service-single-region/"
 resource_group_location = "eastus"
 name = "az-simple"
 deployment_targets = [{
   app_name = "cobalt-backend-api",
   image_name = "msftcse/az-service-single-region",
   image_release_tag_prefix = "release"
 }]
 acr_build_git_source_url = "https://github.com/erikschlegel/echo-server.git"
}

@iphilpot @awkwardindustries ... here are some notes that came from my conversation with @aflinchb (Anne). We reviewed the version & go idea, especially from an App Dev Experience perspective.

where do production/devint/etc secrets live?
where would the env var to point apps (e.g. helium) at the KV get set?
clearly show how different env targets are set by AZ DO (prod vs devint)?
app team prefers to docker build & push images themselves, how does that work? (the az-single-region template is performing a build/push itself)
how do 'ops teams' react to updates at microsoft/cobalt? what might that look like? (and how does upgrades to TF work?)
how does Cobalt's position on IaC cover the lifecycle of CosmosDB? whitelisted IP? note the ISO template, as an example
describe the 'anotomy' of everything (repos, versions, artifacts, use cases, personas)
enable turning off (Cobalt's) integration test for downstream users of a template

This work is finished. Closing

microsoft / cobalt