jordanh
commented
4 years ago Quick update to capture current state and evolve thinking. After needing to add additional capacity we've bumped the priority of v4 - break out graphql execution service to now so that we can more easily take advantage of multi-core execution, even if we remain hosted on a single blade.
Open questions for discussion:
- Do we host database services inside or outside of k8s? (we are leaning toward in, using mounted volumes and stateful sets)
- Do we use a managed k8s service? (I am leaning toward yes)
- Do we define our architecture using Terraform? (I am leaning toward yes)
- How do we store key material? (I am leaning toward HashiCorp Vault)
- How are DB backups managed (possibly: a separately tagged rethinkdb replica set, both for backups and BI queries)?
- How do we manage managed private deployments?
Resketching the above:
v1 - achieve dokku feature parity
create a helm chart of parabol
- nginx ingress, cert-managed with let's encrypt
- service/deployment for web, 1 replica. PM2 removed
- service/deployment for redis, 1 replica
- service/deployment for rethinkdb, 1 repllica
- ready & liveliness listeners on all 3 via datadog agent checks
- 1 Job that serves as the postdeploy hook
- Prometheus logging piped to datadog
- github actions used as CI/CD
- 2 versions released with breaking DB migrations while prod still points to current deployment
- 1 control plane for prod, 1 for staging. additional dev envs can be created in the staging control plane by using namespaces. staging control plane can be shut down when not in use.
- deprecate action-chronos in favor of CronJobs (https://medium.com/@ksholla20/creating-a-cronjob-in-kubernetes-and-use-secrets-as-environment-variables-b8e0166f2d51)
- Where to deploy? shouldn't matter too much. still tbd, wherever is cheapest, probably.
v2 - cluster rethinkdb, redis; realtime data backup
- redis lock supports a redis cluster
- presence moved to redis #3259
- rate limiter moved to redis
- second rethinkdb cluster added via 2nd server tag for realtime replication, periodic cold-store backup, and BI
v3 - cluster web, database
- full server audit for possible race conditions
- web app replicas scaled to 3
- database sharding and replication used for production tag
carloslfu
Been deep into k8s for the last couple days. Here's what a 3-month plan looks like. I feel comfortable doing it all, but our server is healthy as can be & can handle a LOT more traffic, so no real tension to fix what isn't broken.
v1 - achieve dokku feature parity
v2 - cluster rethinkdb, redis
v3 - cluster web
v4 - break out graphql execution service
v5 - TURN server & mediasoup deployments added
On Premise