search

monogon-dev / monogon

The Monogon Monorepo. May contain traces of peanuts and a ✨pure Go Linux userland✨. Work in progress!
https://monogon.tech
Apache License 2.0
378 stars 8 forks source link

Metropolis Control Plane HA #261

Open q3k opened 11 months ago

q3k commented 11 months ago

We've never really tested Control Plane HA in an end-to-end scenario, and we reaped the effects of that during our first large production deploying.

Related fixes:

  1. https://review.monogon.dev/c/monogon/+/2067
  2. https://review.monogon.dev/c/monogon/+/2068
  3. https://review.monogon.dev/c/monogon/+/2069
  4. https://review.monogon.dev/c/monogon/+/2071

E2E tests are still in progress.

q3k commented 10 months ago

Forensics data from a recent HA control plane failure:

https://drive.google.com/drive/folders/1knvTErYqXKEXUyCxvaNpHyM4tikz7QXY