Bedrock's k8s based demo website deployments

[duallain]

Goal - To make it so devs working on bedrock have the ability to spin up demo websites for specific versions of bedrock.

Requirements - (from the POV of users)

• A push to demo/* creates a working bedrock website
• That website must have valid dns names, and working certs
• That website must be the version of the code on that branch
• There ought to be a slack notification for state changes (start/stop/update)
• On branch deletion the website should be deleted
• Hide most implementation details/document the interface
• Provide an interface to (optionally?) run functional tests against that website

From the POV of the infra -

• One alb per service (where bedrock is the 'service')
• gitlab pipeline creates the needed resources for each demo website
• kubectl apply does all the 'work' to create the resources. (no lambda/cron job type thing needed)

Implementation Intentions - (this is a best guess about how to do the work, but flexible) Once per cluster do -

• route53 domain for bedrock demos will be (X) (@metadave can you help me pick here?)
• Create an iam policy that can create entries in that domain
  • should have a pipeline
  • should be stored in code
• setup cluster level services in aws k8s clusters
  • voyager to manage alb/ingress
  • probably helm install
  • should be stored as code
  • create a pipeline to deploy this
  • external dns to manage dns entries
  • probably helm install
  • should be stored as code
  • create a pipeline to deploy this
  • kube2iam to allow assuming the created iam policy

Once per service do -

• the combinedIngress job matching demo/*
• document the interface
• write/collect the k8s deployment of teh story somewhere this pipeline can use it
• publish a docker container of the branch
• inject the version of that container into the k8s deployment files

Notes - Reqs from @alexgibson

• git push a branch to deploy a demo.
• slack notification when build starts/ends/fails.
• ability to push to a run-integration-tests branch (or similar), where a branch is deployed and then tests are run against it (slack notifications for tests start/end/fail).
• preserve demo1-5 domain names, as we rely on these for Firefox Account CORS requests, and also translators need URLs that don't expire after X days.

[bookshelfdave]

That website must have valid dns names,

how do we ensure that branch names turn into valid DNS names?

One alb per service

in the past with Voyager, we were able to use wildcard DNS and have all services share a single ELB. I think this is cheaper than dynamically creating new ALBs, which can rack up charges for un-pruned branches.

route53 domain for bedrock demos will be (X) (@metadave can you help me pick here?)

mozmar.org, ramzom.org are fine not moz.works

[pmac]

not moz.works

Why not? The old demos were at a moz.works domain. Doesn't matter to me really, just curious. That also reminds me, do we have mozmeao.org?

[bookshelfdave]

not moz.works Why not?

the Voyager controller automates DNS zone changes, so you have to give it full access to manage moz.works. I don't want Voyager accidentally messing with moz.works.

[pmac]

Also very much agreed about wildcard DNS and Certs if possible. I'll also just go on record one more time to suggest that bedrock is the only service for which we need these arbitrarily named demos, and that this all seems like a lot of work for little reward just so that we can use k8s for this. I'm still personally happier with the Dokku approach which keeps these things both off our prod infra and managed with easy scripts from GitLab. See the changes required here.

[duallain]

That website must have valid dns names,

how do we ensure that branch names turn into valid DNS names?

I think I'll probably just be opinionated about which branch names can build. Add a regex to reject slashes and dots and write that in the docs. Dashes and underscores ought to be ok I think.

One alb per service

in the past with Voyager, we were able to use wildcard DNS and have all services share a single ELB. I think this is cheaper than dynamically creating new ALBs, which can rack up charges for un-pruned branches.

I was thinking that bedrock should have one alb. All demos sites for bedrock can use that same alb. (service == bedrock in this context).

[pmac]

how do we ensure that branch names turn into valid DNS names?

GitLab will do that for us with the CI_COMMIT_REF_SLUG variable.

See docs for reference.

[pmac]

As I used here: https://github.com/mozilla/bedrock/compare/demo/dokku#diff-e23facd71a79c9a4069ba66ec2f6afeeR13

[glogiotatidis]

It feels that there's already a lot of work purred into implementing this in k8s and while close it's not ready yet, if I understand this correctly. I'll echo @pmac that this is only for bedrock. If it can be made into a generic k8s component maybe it's worth the extra time and effort.

combinedIngress code looks good btw.

[duallain]

I agree with you two. Too much work for just demo sites for sure. I think we should be able to reuse voyager/external dns with the full deployments of all services that are public facing. There's a bit of code somewhere in the infra-private repo (dns/aws/moz.works.tf) that we might be able to get rid of with this work. We'll also be able to upgrade to albs (which I've found to be a tiny bit more stable) and share the albs between bedrock/basket deployments (to save $$).

[duallain]

work required to finish everything up:

• todo: override env var file example, document and make it work for demos
• ensure named www demo sites continue working as is
• get a review from @alexgibson
• write some release notes and ask a few front end devs to test drive from those notes