What are the best Kafka-Server setups ?

[ddubbert]

There are a couple of different possibilities for setting up brokers with Kafka. The prototype of this game will depend on a single machine to create all services (it is just a first test), so the best settings should be identified for such a single machine approach. Having multiple partitions allows for more parallelism in processing events, but comes with more replication latency, rebalances and open server files.

Therefore PoCs with different setups should be made. The main aspect should be comparing the time needed for an equal set of events to be transmitted. Focussed settings should be the amount of brokers, the amount of consumers, the acknowledgment setting of a post (producer), partition amount and the replication amount. Keep in mind that there is only one machine available, so these tests are only meaningful for a one node setup where the machine has following specs:

• Processor: 2,6 GHz 6-Core Intel Core i7
• Memory: 32 GB 2400 MHz DDR4
• Storage: 500GB SSD, One Partition

[ddubbert]

Different PoCs have been created and their time for sending and receiving 20000 events with a maximum size of 32-Bytes have been measured. For all of them an average value has been calculated out of 5 tries.

Single Broker and Single Consumer: (Partitions 1 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 6882, 6242, 6312, 6133, 6194
  • Average: 6352.6

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 6381, 5863, 5889, 5786, 5928
  • Average: 5969.4

• Acknowledgment 0 (none):

  • Milliseconds: 2394, 2337, 2349, 2401, 2337
  • Average: 2363.6

Single Broker and Single Consumer: (Partitions 3 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 6665, 7021, 6554, 6442, 6480
  • Average: 6632.4

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 6274, 6331, 6110, 6096, 6158
  • Average: 6193.8

• Acknowledgment 0 (none):

  • Milliseconds: 2539, 2768, 2707, 2657, 2494
  • Average: 2633

Single Broker and Tripple Consumer: (Partitions 3 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 12859, 12302, 11227, 11327, 11680
  • Average: 11879

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 11252, 11026, 10906, 10903, 10877
  • Average: 10992.8

• Acknowledgment 0 (none):

  • Milliseconds: 4966, 4366, 3725, 3323, 3558
  • Average: 3987.6

Tripple Broker and Single Consumer: (Partitions 3 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 8945, 8569, 8715, 8610, 8456
  • Average: 8659

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 8618, 8888, 8302, 8491, 8240
  • Average: 8507.8

• Acknowledgment 0 (none):

  • Milliseconds: 2423, 2439, 2860, 2554, 2506
  • Average: 2556.4

Tripple Broker and Single Consumer: (Partitions 3 & Replica 2)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 21045, 20263, 24996, 24812, 24636
  • Average: 23150.4

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 14828, 14042, 14096, 13776, 13673
  • Average: 14083

• Acknowledgment 0 (none):

  • Milliseconds: 8493, 5270, 5634, 5264, 6040
  • Average: 6140.2

Tripple Broker and Tripple Consumer: (Partitions 3 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 13130, 12626, 12913, 12506, 12462
  • Average: 12727.4

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 13442, 12762, 12514, 12307, 12443
  • Average: 12693.6

• Acknowledgment 0 (none):

  • Milliseconds: 4531, 4292, 4178, 4755, 4624
  • Average: 4476

Tripple Broker and Tripple Consumer: (Partitions 3 & Replica 2)

• Acknowledgment -1 (all Partitions):

  • Milliseconds: 19748, 20431, 19731, 19524, 19480
  • Average: 19782.8

• Acknowledgment 1 (only leader Partition):

  • Milliseconds: 13870, 13668, 13413, 13506, 13219
  • Average: 13535.2

• Acknowledgment 0 (none):

  • Milliseconds: 5097, 4907, 4968, 5041, 4997
  • Average: 5002

Single Broker and Tripple Topic: (Partitions 1 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds (For all Topics): [11126 11175 11155], [10689 10773 10781], [10070 10050 10117]
  • Average: 10659.5

• Acknowledgment 0 (none):

  • Milliseconds (For all Topics): [4978 5026 4970], [4283 4125 4332], [4254 4084 4208]
  • Average: 4473.3
  • Notes: Events aren't always ordered the way they should be.

Tripple Broker and Tripple Topic: (Partitions 1 & Replica 1)

• Acknowledgment -1 (all Partitions):

  • Milliseconds (For all Topics): [14756 14533 14375], [14384 14203 13980], [14215 13999 13974]
  • Average: 14268.7

• Acknowledgment 0 (none):

  • Milliseconds (For all Topics): [7046 6910 7038], [4816 4882 4862], [4865 4862 4785]
  • Average: 5562.8
  • Notes: Events aren't always ordered the way they should be. Doesn't happen as often as with a single broker.

[ddubbert]

Review:

• Amount Broker: Using a single Broker overall had the best performance. Adding more Broker reduced the performance slightly, which can be caused by all of them trying to write and read from the same disk (see this Stackoverflow comment) and simply having more processes on one machine in general. For a single machine and the creation of a prototype, using a single Broker might be best. If the project will be released, adding more Brokers on different Nodes will be the best option because of reliability. This benchmark also notes, that a single broker with a single partition is able to handle 10s of MB/sec, which should be sufficient for this project.

• Amount Partitions: The amount of Partitions doesn't influence the performance by much. More Partitions are a precondition for parallel processing of events which can in turn improve the performance when consumers do more than just print an event. A Partition amount of 3 should be sufficient for a first prototype. The amount can be raised when more than three consumers are needed. When scaling, the maximum amount of partitions per Broker can found in this test / article. It is: 100 b r, where b is the Broker amount and r the amount of Replicas. This article also gives a lot of information about using multiple Partitions.

• Amount Replicas: Replicating partitions is only possible with multiple brokers / servers. Raising the amount of Replicas also raises the time needed for processing an event by quite much. Replicas are one of the biggest factors for performance reduction in this test. Replicas are supposed to improve reliability in improving data availability. In this prototype every process will be run on a single machine, so Replicas wont have any benefit (if the machine crashes, every process will be ended). In addition, a single Broker will be used first, so Replicas aren't possible. Lastly the events of this project should be read immediately or never, because they will be outdated after a short period of time. Thats why no Replicas are needed.

• Amount Consumer: Having multiple consumer in a consumer group is only viable if there are multiple partitions. Only in this case they are able to work in parallel and with that have the chance to improve the performance. In this test, using a single Broker and multiple Consumer reduced the performance by a bit but using multiple of both raised the performance. Those Consumers didn't do much of data processing which might have interfered with the performance, but when they do more of processing, it might / will interfere with it. The implementation of the Prototype will start with a single Broker and should therefore also start with a single Consumer per Topic. When performance issues appear, other Consumers can be added and if that isn't sufficient, other Brokers should be added as well (only if multiple Nodes are available).

• Acknowledgement: The acknowledgement factor in this test was the most influential factor. Using no acknowledgement improved the performance by very much, even more when multiple Brokers and higher Replica factors where used, but could also end in changed orders of events. Only waiting for the leader already costed a couple of seconds, which was worsened when waiting for all Replicas. In this project, events are short lived and producing new events is much more valuable then resending an event, that didn't get acknowledged. Thats why no acknowledgement should be used. If changes of the order of events happen to often and worsen the game experience, only the acknowledgement of the leader should be awaited to keep some performance benefits.

• Amount Topics: Raising the amount of Topics and Events overall raised the time needed for transmission, mostly caused by increasing the Event amount. If different Events are needed, using Topics in parallel for those Events has a benefit for the performance compared to sending all of the Events through one Topic. In the tests, having three Topics with 20000 events each only doubled the time needed for a single topic with 20000 events, even though the amount of events were tripled. Having multiple Topics on a single Broker also has been more effective than using three Brokers. The prototype should therefore start with all Topics on a single Broker. The amount of Brokers can be increased if performance issues appear and / or multiple Nodes are available. All in all, different event-types should be transmitted in their own Topics, but the amount of Topics should be kept as low as really needed.