search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
128 stars 1.04k forks source link

can we customize the ksql migration topic Partition? #8031

Open besthanliu opened 3 years ago

besthanliu commented 3 years ago

can we customize the ksql migration topic Partition? currently the migration topic Partition is 1 and can not be change.then there have a issue,when we excute migration script in ksql cluster.will fail if we do not config ksql.streams.num.standby.replicas.

rodesai commented 3 years ago

@vcrfxia I'm moving this over to KSE since I think he's asking about query upgrades. LMK if that's not the case and we can pick it back up.

vcrfxia commented 3 years ago

@besthanliu The migrations metadata stream and table must have a single partition for the same reason that the ksqlDB cluster's command topic must only have one partition -- a strong ordering for all messages must be guaranteed. Can you please clarify the problem that you are hitting? It sounds like it might have to do with the number of replicas for the topic, not the number of partitions. The number of replicas is configurable: https://docs.ksqldb.io/en/latest/reference/migrations-tool-configuration/#ksqlmigrationstopicreplicas