Kafka: A Modern Distributed System

[ganler]

I used to use Kafka MQ in my ByteDance internship, but I only got some rudimentary ideas about concepts like "topic", "broker", things like that.

Official Introduction by Tim: https://kafka.apache.org/intro QCon by Tim: https://www.youtube.com/watch?v=Ea3aoACnbEk

[ganler]

Big Idea

Before Kafka: Think as "Things"

Information is usually stored in database only. (we store things with states in the database)

Kafka: "Think as "Events" instead"

Kafka encourages people to think about EVENTs first, and THINGs second.

Events happened: written to the LOG.

• Easier to think.
• Easier to scale.

Kafka manages the states of events using a term called TOPIC, which is an ordered collection of events stored in the DURABLE way.

Here "durable" means things are store in a distributed fashion, with replicas.

"Back to when DBs rule the world", the trend is to build one large program using a big database. Now the trend is to write lots of small logs to small programs that are small enough for users to think. Each of the small programs talks to each other to using messages from a topic(operations are like: consume, produce).

[ganler]

Kafka Basics

Not a queue, but a log(stored persistently).

Events are modeled in K/V pairs, which live in the topic for a given elapsed time(a sys config parameter, say 7 days).

For scalability and fault tolerance, the log is partitioned on many servers. (Say a topic is made of 4 partitions(in the file form)). And the "servers" here are "brokers".

Problems 1, which partition should I select?

The answer is via consistent hashing.

Since we got a hashed value x, x % N(= 4 here) is the destination.

Producer => Clients Consumer => Topics

Replication

For fault tolerance.

Say we'll replica partitions using a replication factor: say 3. (3 replicas)

Leader: One of the replicas is the leader. (If u read, u read the leader first) Followers: Non-leaders.

• Writes always go to the lead replica.
• Followers ask the leader for new writes.
• Reads always* come from the leader. (not for new versions, where "observers" are introduced)

  Leader Election

If the leader died, we'll elect a new leader.

Consumer Groups

A group of consumer clients.

The controller

One broker in Kafka's cluster is elected as a controller.

• Monitors the health of all other brokers.
• Coordinates partition leader election on broker failover.
• Communicates partition leader changes to other brokers.