search

manuzhang / read-it-now

Don't read it later; read it now
3 stars 0 forks source link

Apache Kafka Supports 200K Partitions Per Cluster #32

Open manuzhang opened 5 years ago

manuzhang commented 5 years ago

As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions.

Kafka 1.1.0 has made significantly improvement in the number of partitions a single Kafka cluster can support while preserving high availability and durability. The improvement mainly comes from the reduced controlled shutdown time (from 6.5 minutes to 3 seconds as per tests) thanks to

  1. fixing a logging overhead
  2. using the asynchronous API when writing to Zookeeper
  3. batching communication of new leaders

There is also big improvement in controller failover time using the asynchronous API to reload state of all partitions from Zookeeper.

The improvement that we made in 1.1.0 is just one step towards our ultimate goal of making Kafka infinitely scalable.