Creating a changefeed on a multi-family table with the split_column_families attempts to create a topic called "{family}" on the broker, despite this not being the name of any column family defined in the table.
To Reproduce
Create a new cluster, single node is enough. Have a downstream broker such as Kafka/Redpanda with automatic topic creation enabled.
Run the following SQL:
CREATE TABLE t (
id UUID PRIMARY KEY,
s STRING,
FAMILY "a" (id),
FAMILY "b" (s)); -- two column families, called a and b
CREATE CHANGEFEED FOR TABLE t INTO '<kafka sink address>' WITH OPTIONS (split_column_families);
Without populating the table with any entries, if I go and look at the created topics, I see one that shouldn't be there:
NAME PARTITIONS REPLICAS
t._u007b_family_u007d_ 1 1
This topic does not show up on 23.2.x, but it does show up in 24.1.x and 24.2.x. However, all versions (including 23.2.x) show this nonsense notice message:
NOTICE: changefeed will emit to topic t._u007b_family_u007d_
With a broker that doesn't automatically create topics, the changefeed cannot start up and produces the following log message:
Changefeed job <job id> encountered transient error: ‹kafka server: Request was for a topic or partition that does not exist on this broker› (attempt 1)
Expected behavior
It should not attempt to create a {_family_} topic, especially as this topic doesn't receive any messages. This breaks changefeeds that send to a broker that has disabled automatic topic creation as they cannot start up. Also the above NOTICE message should not be produced.
Environment:
CockroachDB version: 23.2.x (for the NOTICE message), 24.1.x and 24.2.x for the NOTICE and message creation.
Describe the problem
Creating a changefeed on a multi-family table with the
split_column_families
attempts to create a topic called "{family}" on the broker, despite this not being the name of any column family defined in the table.To Reproduce
Create a new cluster, single node is enough. Have a downstream broker such as Kafka/Redpanda with automatic topic creation enabled.
Run the following SQL:
Without populating the table with any entries, if I go and look at the created topics, I see one that shouldn't be there:
This topic does not show up on 23.2.x, but it does show up in 24.1.x and 24.2.x. However, all versions (including 23.2.x) show this nonsense notice message:
With a broker that doesn't automatically create topics, the changefeed cannot start up and produces the following log message:
Expected behavior
It should not attempt to create a
{_family_}
topic, especially as this topic doesn't receive any messages. This breaks changefeeds that send to a broker that has disabled automatic topic creation as they cannot start up. Also the above NOTICE message should not be produced.Environment:
Jira issue: CRDB-41314