edx-platform deployment pipeline/monitoring training

[davidjoy]

A/C

• [ ] Proposal for how to split edx-platform pipeline ownership to be distributed to engineers and engineering managers
  • [ ] Distribute said proposal

We want other squads to be able to help maintain and monitor the edx-platform deployment pipeline in order to reduce our on-call burden and upskill our colleagues.

This probably looks like:

• Training sessions on how the deployment pipeline works
• Make sure runbooks are up to date and socialized
• Creating a new on-call rotation that loops in engineers from other teams over time.
• Likely some work to re-think what split-ownership of edx-platform means

[robrap]

Additional thoughts:

1. Who determines if the pipeline is fast enough and would invest in speeding it up?
2. Who determines if the pipeline is stable enough and who would invest in improving flakiness?
3. How do we determine common issues? Ideally this would be automated, and would not rely on various teams following some manual process for every failure to catalog issues.

Note that these issues are bad with a single team, and will become worse with split-ownership, so it would be best if we could improve for everyone before we split.

[robrap]

It's becoming clear to me that our current runbook has some deficiencies:

1. Should the first step simple be to rerun any failed stage, given how many flaky issues we see?
2. At what point do we ticket a flaky issue?
3. How do we gather data on each flaky issue to know in what order they should be addressed? Which are happening the most in recent history?

[robrap]

Waiting on feedback or a Parking Lot discussion around maintenance.

[robrap]

After discussion in Parking Lot, we wondered:

1. Do we need to improve stability of the pipeline before even trying this?
2. Is moving on to ArgoCD a prerequisite for minimizing training on a different pipeline for other teams (if the pipeline becomes more like others)?
3. Is this really worth having split ownership from a 2U perspective?

I'm marking blocked for now, but we may potentially close this.

If we close this ticket, we should:

• [ ] Archive proposal in Confluence.

@jmbowman: Should this remain blocked or should be close?

[robrap]

We are moving this to the backlog until edx-platform is containerized, we've moved on to ArgoCD, and we can reassess then?