Rebalance does not spread to all available brokers

[petedannemann]

We have an unbalanced topic spread across 26/39 available brokers

$ topicctl get balance my-topic-name
Broker replicas:
-----+------------+------------+------------+------------+--------
  ID |    RACK    | POSITION 1 | POSITION 2 | POSITION 3 | TOTAL
-----+------------+------------+------------+------------+--------
  1  | us-west-2c | 0          | 0          | 1          | 1
  2  | us-west-2a | 0          | 1          | 0          | 1
  3  | us-west-2b | 0          | 1          | 0          | 1
  4  | us-west-2c | 0          | 0          | 0          | 0
  5  | us-west-2b | 0          | 0          | 1          | 1
  6  | us-west-2a | 0          | 0          | 1          | 1
  7  | us-west-2a | 0          | 0          | 1          | 1
  8  | us-west-2a | 1          | 1          | 0          | 2
  9  | us-west-2c | 1          | 0          | 0          | 1
  10 | us-west-2c | 1          | 0          | 0          | 1
  11 | us-west-2c | 1          | 1          | 0          | 2
  12 | us-west-2b | 0          | 0          | 0          | 0
  13 | us-west-2b | 1          | 0          | 0          | 1
  14 | us-west-2a | 1          | 0          | 0          | 1
  15 | us-west-2a | 1          | 0          | 0          | 1
  16 | us-west-2b | 1          | 0          | 0          | 1
  17 | us-west-2b | 1          | 0          | 1          | 2
  18 | us-west-2c | 1          | 0          | 1          | 2
  19 | us-west-2c | 0          | 1          | 0          | 1
  20 | us-west-2c | 0          | 0          | 0          | 0
  21 | us-west-2b | 0          | 0          | 0          | 0
  22 | us-west-2a | 0          | 0          | 1          | 1
  23 | us-west-2c | 0          | 0          | 1          | 1
  24 | us-west-2b | 0          | 0          | 0          | 0
  25 | us-west-2c | 0          | 1          | 0          | 1
  26 | us-west-2a | 0          | 0          | 0          | 0
  27 | us-west-2b | 0          | 1          | 0          | 1
  28 | us-west-2a | 0          | 0          | 0          | 0
  29 | us-west-2c | 0          | 1          | 0          | 1
  30 | us-west-2b | 0          | 0          | 0          | 0
  31 | us-west-2a | 0          | 0          | 0          | 0
  32 | us-west-2b | 0          | 0          | 1          | 1
  33 | us-west-2a | 0          | 0          | 0          | 0
  34 | us-west-2a | 0          | 0          | 0          | 0
  35 | us-west-2c | 0          | 0          | 0          | 0
  36 | us-west-2a | 0          | 1          | 0          | 1
  37 | us-west-2b | 0          | 1          | 0          | 1
  38 | us-west-2b | 0          | 0          | 0          | 0
  39 | us-west-2c | 0          | 0          | 1          | 1
-----+------------+------------+------------+------------+--------
> Broker rack replicas:
-------------+------------+------------+------------+--------
     RACK    | POSITION 1 | POSITION 2 | POSITION 3 | TOTAL
-------------+------------+------------+------------+--------
  us-west-2a | 3          | 3          | 3          | 9
  us-west-2b | 3          | 3          | 3          | 9
  us-west-2c | 4          | 4          | 4          | 12
-------------+------------+------------+------------+--------
>>>

The rebalance plan shows the partitions not being spread to all brokers available, which seems unexpected.

$ topicctl apply my-topic-name --dry-run --rebalance
------------+---------------------------------------------------+---------------------------------------------------+-------+----------
[2022-07-29 16:50:24]  INFO Here are the number of partitions per broker now, during the migration, and after:
---------+------------+------------+-------------
  BROKER |    CURR    |    MAX     |  PROPOSED
         | PARTITIONS | PARTITIONS | PARTITIONS
---------+------------+------------+-------------
  1      | 1          | 1          | 1
  2      | 1          | 1          | 1
  3      | 1          | 1          | 1
  4      | 0          | 1 (+1)     | 1 (+1)
  5      | 1          | 1          | 1
  6      | 1          | 1          | 1
  7      | 1          | 1          | 1
  8      | 2          | 2          | 1 (-1)
  9      | 1          | 1          | 1
  10     | 1          | 1          | 1
  11     | 2          | 2          | 1 (-1)
  12     | 0          | 1 (+1)     | 1 (+1)
  13     | 1          | 1          | 1
  14     | 1          | 1          | 1
  15     | 1          | 1          | 1
  16     | 1          | 1          | 1
  17     | 2          | 2          | 1 (-1)
  18     | 2          | 2          | 1 (-1)
  19     | 1          | 1          | 1
  20     | 0          | 0          | 0
  21     | 0          | 0          | 0
  22     | 1          | 1          | 1
  23     | 1          | 1          | 1
  24     | 0          | 0          | 0
  25     | 1          | 1          | 1
  26     | 0          | 0          | 0
  27     | 1          | 1          | 1
  28     | 0          | 0          | 0
  29     | 1          | 1          | 1
  30     | 0          | 0          | 0
  31     | 0          | 0          | 0
  32     | 1          | 1          | 1
  33     | 0          | 1 (+1)     | 1 (+1)
  34     | 0          | 0          | 0
  35     | 0          | 1 (+1)     | 1 (+1)
  36     | 1          | 1          | 1
  37     | 1          | 1          | 1
  38     | 0          | 0          | 0
  39     | 1          | 1          | 1

This seems like some bug in the rebalancing code. Note that if we run multiple rebalances, it will eventually spread the partitions across all brokers available.

[yolken]

It looks like you have 39 brokers but only 30 partition replicas, so I don't think it's possible for your topic to be spread across all brokers at the same time? Let me know if I'm misunderstanding something here.

[petedannemann]

Oops, yes you are absolutely right. I will find an example that actually shows the issue I was trying to highlight and update this issue