Updating topic partitions and replicas of existing topic .

[mussa572]

Hi , Just wanted to confirm if Topic operator support updating the partition and replicas of existing topic which has some data . Here is why I have raised the question .

1) Created the topic using topic operator with 3 partitions and 1 replica . 2) Added some data using the kafka console producer . 3) Updated the topic operator to increase the replicas . 4)Noticed the change on the topic operator . However when seeing on the kafka describe command i didnt see any changes . 5)updated the topic operator to add 10 partitions and 3 replicas . I can see the it updated on the topic operator output . 6) However I didnt see this update on Kafka side .

is it a bug or am I missing something here . As per the documentation we can update the partitions .

[scholzj]

@tombentley @stanlyDoge Could you have a look at this? You understand TO better than me.

[tombentley]

@mussa572 in general you should check that the KafkaTopic.status.conditions[0] becomes Ready after you make a change, in order to be sure that the TO has been able to make the changes.

The TO is not able to change the #replicas because doing that would require it to decide which brokers should host the new replicas and it's not in a position to make an informed choice. So if you need to do that you should make the changes on the Kafka side, and the TO will reflect them on the KafkaTopic resource. If you try to change the replicas you should find that the KafkaTopic.status.conditions[0] becomes NotReady with an explanation.

So if you revert the change in replicas you should find the TO is able to increase the #partitions for you and the status becomes Ready. Then you can make the change to the number of replicas on the Kafka side.

[mussa572]

@tombentley Thanks very much for the reply . Just wonder in where/how I can see the state of the KafkaTopic.status.conditions[0] . Also would like to clarify what changes can we make to the existing topic with TO apart from updating the partitions and replicas ?

[tombentley]

kubectl get kafkatopics -o yaml <your-topic>, I'm using KafkaTopic.status.conditions[0] as a path notation, so look for the status object in the output, then the conditions in that.

You can:

• change anything in spec.config
• increase spec.partitions (Kafka itself prevents this being decreased).

You can't:

• decrease spec.partitions
• change spec.replicas.

We might get around to supporting changes to spec.replicas once we've got "cluster balancing" via CruiseControl working (which is a feature being worked on currently).

[scholzj]

I will close this since there was no update for more than 10 days. If you have something more feel free to open a new issue, reopen this one or get in touch with us on Slack or mailing list.