connector ignores ddl change in clickhouse

[tellienne]

Hello! Thanks for your work on the connector.

We use version 2.3.1-lt to transfer data from postgres to clickhouse. We encountered the problem that sometimes the connector does not see that the table in the clickhouse has changed. Changes are applied only after a restart.

How to play:

1. Add a field to the table in Postgres. In our example, the passing_price field.
2. Using the request, add a field to the clickhouse

ALTER TABLE test.moderated_request_products ON CLUSTER '{cluster}' ADD COLUMN IF NOT EXISTS passing_price Nullable(DECIMAL(14,2)) AFTER updated_at;

3. using ad hoc snapshot, run a snapshot of the table with this field We see in the logs:

2024-11-20 08:43:24.202 INFO - Requested 'BLOCKING' snapshot of data collections '[public.moderated_request_products]' with additional conditions '[]' and surrogate key 'PK of table will be used'

2024-11-20 08:43:24.506 INFO - For table 'public.moderated_request_products' using select statement: 'SELECT "id", "moderated_request_id", "product_id", "quantity", "created_at", "updated_at", "passing_price" FROM "public"."moderated_request_products"'

The connector successfully sees the new field in Postgres.
Then he inserts into the clickhouse, but does not see the new field in the clickhouse

2024-11-20 08:43:25.214 INFO - INSERT QUERY for Database(test) : insert into moderated_request_products(id,moderated_request_id,product_id,quantity,created_at,updated_at,_version,_sign) select id,moderated_request_id,product_id,quantity,created_at,updated_at,_version,_sign from input('id UUID,moderated_request_id UUID,product_id UUID,quantity Decimal(16, 4),created_at String,updated_at Nullable(String),_version UInt64,_sign UInt8')

4. restart the connector
5. after the restart, the connector will see a new field in the clickhouse

The problem is not always reproduced. Sometimes it correctly sees new fields in the clickhouse, sometimes not. We have not yet been able to find a connection. 
Any ideas are welcome

[subkanthi]

Hi @tellienne , unfortunately the current way is to restart , you can use sink-connector-client to restart, we are planning to add an option in the client to refresh the schema https://github.com/Altinity/clickhouse-sink-connector/issues/891