Implement configuration consumer

[evanpurkhiser]

[!NOTE] Original design document section

For uptime monitoring multiple domains we need some way to tell the configuration about each of checks it needs to be making. We will do this through a CheckConfiguration each check configuration will map back to a subscription_id in sentry (similar to a monitor_environment in crons).

We need a way to propagate these configurations from sentry to each of the checkers (there may be N checkers).

Approach

We will produce configuration messages into a kafka topic uptime-configurations. This topic will be used as persistent storage for configurations. See Is it ok to store data in Kafka?.

This topic will receive configurations as they are created / re-configured via sentry. The topic will have an indefinite retention window and will use Log Compaction to clear out older configuration messages in favor of the most recently produced configuration message. Tombstones for configurations that have been removed will allow for deletion of configurations that are no longer needed.

The consumer that is part of each uptime-checker will work by reading all configuration messages at boot. Importantly, consumers will never commit offsets. Each time a checker boots, it will read the configurations for the partitions it is assigned to. This does mean a checker will need some time to boot.

[!NOTE] We profiled this: we can read 1m configs (that look similar to what production configs may look like) from a single partition in a Kafka topic in 13.5s

Advantages to this approach

• It’s easy to have multiple schedulers. We would partition the config topic via the key to guarantee that all updates for a particular uptime monitor go to the same scheduler.

• Updates are received in real time (no polling), so we can update checks more quickly

• No need to fetch huge numbers of configs from an api in Sentry, and no need to worry about ttls, stale configs and refetching.

• Rebalancing is useful. We will have multiple schedulers, and if one scheduler goes down then Kafka will rebalance those partitions to the remaining consumers. If we used a database storage method we'd need to figure out how to partition work, and what to do when a worker goes down.

Disadvantages

• We can’t easily repartition Kafka here, since we’re relying on all configs belonging to the same partition. A more expensive migration process would be necessary to rebalance. It seems unlikely that this will become a problem for us any time in the next few years though. We can also choose a high number of partitions to mitigate this

Producing configurations from sentry

We will use an Outbox approach to ensure uptime configurations are written to Kafka. This is necessary to ensure eventual consistency of configurations. This will also be used for deletions for creating tombstones that will be compacted away.

[fpacifici]

A trade off to keep in mind regarding restarts (at every deployment). When we restart a Kubernetes deployment we do a rolling restart. There are good reasons to avoid a scenario where we stop the previous version and only then start the new version: if the new version does not start we are down. Blue green is hard to achieve in Kafka as the new consumers trigger rebalancing as soon as they are up.

In a nutshell, in Kubernetes this means that the new pods are started, when the pod's readiness check passes (this is controllable by the application) K8s will start directing traffic to the pod, if the pod responds to requests - Kafka is different as it is a poll system. Once enough pods are ready K8s sends a SIGTERM to the old ones which terminate cleanly.

The Kafka approach:

• the rolling restart triggers a rebalance of partitions each time a pod joins or one leave. This means that for a period of time partitions are being reshuffled.
• This system will load the entire configuration only when all the rolling restart is done. Before that the pods are mostly inactive.
• So after the rolling restart is complete the system would still not be doing any useful work till the configuration loading is complete.
• This is a risk you have to take into account: for a period of time that can vary and can reach a minute or so your consumer does nothing. You do not really control the overall duration of this period of time because it depends on Kafka, on the K8s scheduler, on GKE, etc.

The K8s deployment approach behind a Service:

• Here your config is provided to the pods by either sending an HTTP call or via periodic refresh.
• The deployment is simpler: as soon as a new pod is running it loads the config then it can make the readiness probe pass and it can start doing work.
• The downside is that pods do not have an identity, As you want to split the space of monitors that each pod manages, you need some sort of consensus on who takes care of which partitions (which in Kafka is managed by partition assignment). This can get complicated

The statefulset approach behind a Service:

• Here your config is provided to the pods by either sending an HTTP call or via periodic refresh. The pods are part of a stateful set.
• The stateful set manages the assignments for you. Pods have an identity so they can be configured to pick up a specific partition of the monitors, also k8s guarantees (well, unless things break) that only one pod is running for a given identity so you do not have double assignments.
• The problem is scaling up/down as that operations would have to trigger a reassignment thus you cannot simply add pods. You end up with the same problem above of having to implement some sort of consensus to dynamically reassign partitions. There is a way around by always restarting the entire deployment if we change the number of replicas if you can accept some duplication of work.