Closed Xuquansheng closed 7 months ago
Did you try to use FINAL in CH or set final=1 ? It is recommended that the read user has this in its user profile, so that duplicates are discarded.
you can also optimize the table (or similar)
@Xuquansheng , please refer to this https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/mutable_data.md on how updates/deletes are handled in sink connector.
Thanks, both aadant and subkanthi, I'm tring to understand how updates/deletes are handled in sink connector.
after i went through the article mentioned, i understood more about updates/deletes in sink connector. but, i still can not know why unchanged 39 records were replicated to clickhouse again and again once i just updated one record on odoo screen.
What about the count with final=1, is it matching ?
yes, final=1 is working, below is message: i just updated id=3, name "Mitchell Admin" to "Mitchell Admin1" on odoo screen:
0eb6d0a88824 :) select id,create_date,name,write_date,_sign,_version from res_partner;
SELECT id, create_date, name, write_date, _sign, _version FROM res_partner
Query id: 920b41f3-40b9-4cec-9425-77afee45ce65
┌─id─┬────────────────create_date─┬─name──────────────────┬─────────────────write_date─┬─_sign─┬──────_version─┐ │ 1 │ 2024-01-25 15:59:45.585470 │ YourCompany │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 2 │ 2024-01-25 15:59:48.138483 │ OdooBot │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 3 │ 2024-01-25 15:59:48.138483 │ Mitchell Admin │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 4 │ 2024-01-25 15:59:48.138483 │ Public user │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 5 │ 2024-01-25 15:59:48.138483 │ Default User Template │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 6 │ 2024-01-25 15:59:48.138483 │ Portal User Template │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 7 │ 2024-01-25 15:59:48.138483 │ Marc Demo │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 8 │ 2024-01-25 15:59:48.138483 │ Joel Willis │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 9 │ 2024-01-25 15:59:48.138483 │ Wood Corner │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 10 │ 2024-01-25 15:59:48.138483 │ Deco Addict │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 11 │ 2024-01-25 15:59:48.138483 │ Gemini Furniture │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 12 │ 2024-01-25 15:59:48.138483 │ Ready Mat │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 13 │ 2024-01-25 15:59:48.138483 │ The Jackson Group │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 14 │ 2024-01-25 15:59:48.138483 │ Azure Interior │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 15 │ 2024-01-25 15:59:48.138483 │ Lumber Inc │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 16 │ 2024-01-25 15:59:48.138483 │ Willie Burke │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 17 │ 2024-01-25 15:59:48.138483 │ Ron Gibson │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 18 │ 2024-01-25 15:59:48.138483 │ Douglas Fletcher │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 19 │ 2024-01-25 15:59:48.138483 │ Floyd Steward │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 20 │ 2024-01-25 15:59:48.138483 │ Edwin Hansen │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 21 │ 2024-01-25 15:59:48.138483 │ Billy Fox │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 22 │ 2024-01-25 15:59:48.138483 │ Jesse Brown │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 23 │ 2024-01-25 15:59:48.138483 │ Soham Palmer │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 24 │ 2024-01-25 15:59:48.138483 │ Kim Snyder │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 25 │ 2024-01-25 15:59:48.138483 │ Edith Sanchez │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 26 │ 2024-01-25 15:59:48.138483 │ Brandon Freeman │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 27 │ 2024-01-25 15:59:48.138483 │ Nicole Ford │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 28 │ 2024-01-25 15:59:48.138483 │ Toni Rhodes │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 29 │ 2024-01-25 15:59:48.138483 │ Gordon Owens │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 30 │ 2024-01-25 15:59:48.138483 │ Theodore Gardner │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 31 │ 2024-01-25 15:59:48.138483 │ Oscar Morgan │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 32 │ 2024-01-25 15:59:48.138483 │ Tom Ruiz │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 33 │ 2024-01-25 15:59:48.138483 │ Colleen Diaz │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 34 │ 2024-01-25 15:59:48.138483 │ Lorraine Douglas │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 35 │ 2024-01-25 15:59:48.138483 │ Addison Olson │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 36 │ 2024-01-25 15:59:48.138483 │ Sandra Neal │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 37 │ 2024-01-25 15:59:48.138483 │ Julie Richards │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 38 │ 2024-01-25 15:59:48.138483 │ Travis Mendoza │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 39 │ 2024-01-25 15:59:48.138483 │ Chester Reed │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 40 │ 2024-01-25 15:59:48.138483 │ Dwayne Newman │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ └────┴────────────────────────────┴───────────────────────┴────────────────────────────┴───────┴───────────────┘ ┌─id─┬────────────────create_date─┬─name──────────────────┬─────────────────write_date─┬─_sign─┬──────_version─┐ │ 1 │ 2024-01-25 15:59:45.585470 │ YourCompany │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 2 │ 2024-01-25 15:59:48.138483 │ OdooBot │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 3 │ 2024-01-25 15:59:48.138483 │ Mitchell Admin1 │ 2024-01-25 16:04:19.783044 │ 1 │ 1706198659898 │ │ 4 │ 2024-01-25 15:59:48.138483 │ Public user │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 5 │ 2024-01-25 15:59:48.138483 │ Default User Template │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 6 │ 2024-01-25 15:59:48.138483 │ Portal User Template │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 7 │ 2024-01-25 15:59:48.138483 │ Marc Demo │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 8 │ 2024-01-25 15:59:48.138483 │ Joel Willis │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 9 │ 2024-01-25 15:59:48.138483 │ Wood Corner │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 10 │ 2024-01-25 15:59:48.138483 │ Deco Addict │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 11 │ 2024-01-25 15:59:48.138483 │ Gemini Furniture │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 12 │ 2024-01-25 15:59:48.138483 │ Ready Mat │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 13 │ 2024-01-25 15:59:48.138483 │ The Jackson Group │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 14 │ 2024-01-25 15:59:48.138483 │ Azure Interior │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 15 │ 2024-01-25 15:59:48.138483 │ Lumber Inc │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 16 │ 2024-01-25 15:59:48.138483 │ Willie Burke │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 17 │ 2024-01-25 15:59:48.138483 │ Ron Gibson │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 18 │ 2024-01-25 15:59:48.138483 │ Douglas Fletcher │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 19 │ 2024-01-25 15:59:48.138483 │ Floyd Steward │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 20 │ 2024-01-25 15:59:48.138483 │ Edwin Hansen │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 21 │ 2024-01-25 15:59:48.138483 │ Billy Fox │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 22 │ 2024-01-25 15:59:48.138483 │ Jesse Brown │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 23 │ 2024-01-25 15:59:48.138483 │ Soham Palmer │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 24 │ 2024-01-25 15:59:48.138483 │ Kim Snyder │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 25 │ 2024-01-25 15:59:48.138483 │ Edith Sanchez │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 26 │ 2024-01-25 15:59:48.138483 │ Brandon Freeman │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 27 │ 2024-01-25 15:59:48.138483 │ Nicole Ford │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 28 │ 2024-01-25 15:59:48.138483 │ Toni Rhodes │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 29 │ 2024-01-25 15:59:48.138483 │ Gordon Owens │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 30 │ 2024-01-25 15:59:48.138483 │ Theodore Gardner │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 31 │ 2024-01-25 15:59:48.138483 │ Oscar Morgan │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 32 │ 2024-01-25 15:59:48.138483 │ Tom Ruiz │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 33 │ 2024-01-25 15:59:48.138483 │ Colleen Diaz │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 34 │ 2024-01-25 15:59:48.138483 │ Lorraine Douglas │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 35 │ 2024-01-25 15:59:48.138483 │ Addison Olson │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 36 │ 2024-01-25 15:59:48.138483 │ Sandra Neal │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 37 │ 2024-01-25 15:59:48.138483 │ Julie Richards │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 38 │ 2024-01-25 15:59:48.138483 │ Travis Mendoza │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 39 │ 2024-01-25 15:59:48.138483 │ Chester Reed │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 40 │ 2024-01-25 15:59:48.138483 │ Dwayne Newman │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ └────┴────────────────────────────┴───────────────────────┴────────────────────────────┴───────┴───────────────┘
80 rows in set. Elapsed: 0.007 sec.
0eb6d0a88824 :) select id,create_date,name,write_date,_sign,_version from res_partner final;
SELECT id, create_date, name, write_date, _sign, _version FROM res_partner FINAL
Query id: a818db75-5457-4b70-8c62-91e4515b0a07
┌─id─┬────────────────create_date─┬─name──────────────────┬─────────────────write_date─┬─_sign─┬──────_version─┐ │ 1 │ 2024-01-25 15:59:45.585470 │ YourCompany │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 2 │ 2024-01-25 15:59:48.138483 │ OdooBot │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 3 │ 2024-01-25 15:59:48.138483 │ Mitchell Admin1 │ 2024-01-25 16:04:19.783044 │ 1 │ 1706198659898 │ │ 4 │ 2024-01-25 15:59:48.138483 │ Public user │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 5 │ 2024-01-25 15:59:48.138483 │ Default User Template │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 6 │ 2024-01-25 15:59:48.138483 │ Portal User Template │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 7 │ 2024-01-25 15:59:48.138483 │ Marc Demo │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 8 │ 2024-01-25 15:59:48.138483 │ Joel Willis │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 9 │ 2024-01-25 15:59:48.138483 │ Wood Corner │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 10 │ 2024-01-25 15:59:48.138483 │ Deco Addict │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 11 │ 2024-01-25 15:59:48.138483 │ Gemini Furniture │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 12 │ 2024-01-25 15:59:48.138483 │ Ready Mat │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 13 │ 2024-01-25 15:59:48.138483 │ The Jackson Group │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 14 │ 2024-01-25 15:59:48.138483 │ Azure Interior │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 15 │ 2024-01-25 15:59:48.138483 │ Lumber Inc │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 16 │ 2024-01-25 15:59:48.138483 │ Willie Burke │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 17 │ 2024-01-25 15:59:48.138483 │ Ron Gibson │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 18 │ 2024-01-25 15:59:48.138483 │ Douglas Fletcher │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 19 │ 2024-01-25 15:59:48.138483 │ Floyd Steward │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 20 │ 2024-01-25 15:59:48.138483 │ Edwin Hansen │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 21 │ 2024-01-25 15:59:48.138483 │ Billy Fox │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 22 │ 2024-01-25 15:59:48.138483 │ Jesse Brown │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 23 │ 2024-01-25 15:59:48.138483 │ Soham Palmer │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 24 │ 2024-01-25 15:59:48.138483 │ Kim Snyder │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 25 │ 2024-01-25 15:59:48.138483 │ Edith Sanchez │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 26 │ 2024-01-25 15:59:48.138483 │ Brandon Freeman │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 27 │ 2024-01-25 15:59:48.138483 │ Nicole Ford │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 28 │ 2024-01-25 15:59:48.138483 │ Toni Rhodes │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 29 │ 2024-01-25 15:59:48.138483 │ Gordon Owens │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 30 │ 2024-01-25 15:59:48.138483 │ Theodore Gardner │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 31 │ 2024-01-25 15:59:48.138483 │ Oscar Morgan │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 32 │ 2024-01-25 15:59:48.138483 │ Tom Ruiz │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 33 │ 2024-01-25 15:59:48.138483 │ Colleen Diaz │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 34 │ 2024-01-25 15:59:48.138483 │ Lorraine Douglas │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 35 │ 2024-01-25 15:59:48.138483 │ Addison Olson │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 36 │ 2024-01-25 15:59:48.138483 │ Sandra Neal │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 37 │ 2024-01-25 15:59:48.138483 │ Julie Richards │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 38 │ 2024-01-25 15:59:48.138483 │ Travis Mendoza │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 39 │ 2024-01-25 15:59:48.138483 │ Chester Reed │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ │ 40 │ 2024-01-25 15:59:48.138483 │ Dwayne Newman │ 2024-01-25 16:01:13.110923 │ 1 │ 1706198527448 │ └────┴────────────────────────────┴───────────────────────┴────────────────────────────┴───────┴───────────────┘
40 rows in set. Elapsed: 0.009 sec.
The issue has been fixed by using altinityinfra/clickhouse-sink-connector:408-97b1d3d83ef93c1b76a2b1c4d9c544dc67fbbec3-lt
I try to use clickhouse-sink-connector to replicate pg database for Odoo to clickhouse. I created demo database on odoo screen. Originally, replica is correct. I don't know how to modify config_postgres.yml to get correct replica. below is message: e55d79b802a9 :) select count(*) from res_partner;
SELECT count(*) FROM res_partner
Query id: a92de383-b5a0-49e1-a42b-ddc5b1b790fc
┌─count()─┐ │ 40 │ └─────────┘
1 row in set. Elapsed: 0.003 sec. I updated one record on odoo screen, got 80 records in table res_partner in clickhouse
e55d79b802a9 :) select count(*) from res_partner;
SELECT count(*) FROM res_partner
Query id: e884ecc1-ae80-4aee-9b86-2304eee797a7
┌─count()─┐ │ 80 │ └─────────┘
1 row in set. Elapsed: 0.006 sec
below is docker-compose-postgres.yml: services: odoo17: restart: always image: odoo:17 depends_on:
"wal_level=logical"
clickhouse: container_name: clickhouse image: clickhouse/clickhouse-server:latest restart: "no" ports:
clickhouse_home:/var/lib/clickhouse
clickhouse-sink-connector-lt: image: altinityinfra/clickhouse-sink-connector:443-ea823c59644963cf752acef3987701a134cb6906-lt entrypoint: ["sh", "-c", "java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Xms4g -Xmx4g -jar /app.jar /config.yml com.alt inity.clickhouse.debezium.embedded.ClickHouseDebeziumEmbeddedApplication"] restart: "no" ports:
volumes: odoo17-conf: odoo17-web: odoo17-addons: odoo17-db: clickhouse_home:
and config_postgres.yml [root@superset docker]# more config_postgres.yml
Some of the properties are part of Debezium PostgreSQL Connector
https://debezium.io/documentation/reference/stable/connectors/postgresql.html
name: Unique name for the connector. Attempting to register again with the same name will fail.
name: "debezium-embedded-postgres"
database.hostname: IP address or hostname of the PostgreSQL database server.
database.hostname: "db"
database.port: Integer port number of the PostgreSQL database server listening for client connections.
database.port: "5432"
database.user: Name of the PostgreSQL database user to be used when connecting to the database.
database.user: "root"
database.password: Password of the PostgreSQL database user to be used when connecting to the database.
database.password: "root"
database.server.name: The name of the PostgreSQL database from which events are to be captured when not using snapshot mode.
database.server.name: "ER54"
schema.include.list: An optional list of regular expressions that match schema names to be monitored;
schema.include.list: public
plugin.name: The name of the PostgreSQL logical decoding plug-in installed on the PostgreSQL server. Supported values are decoderbufs, and pgoutput.
plugin.name: "pgoutput"
table.include.list: An optional list of regular expressions that match fully-qualified table identifiers for tables to be monitored;
table.include.list: "public.tm,public.tm2,public.foo"
table.include.list: "public.*"
clickhouse.server.url: Specify only the hostname of the Clickhouse Server.
clickhouse.server.url: "clickhouse"
clickhouse.server.user: Clickhouse Server User
clickhouse.server.user: "root"
clickhouse.server.password: Clickhouse Server Password
clickhouse.server.password: "root"
clickhouse.server.port: Clickhouse Server Port
clickhouse.server.port: "8123"
clickhouse.server.database: Clickhouse Server Database
clickhouse.server.database: "test"
database.allowPublicKeyRetrieval: "true" https://rmoff.net/2019/10/23/debezium-mysql-v8-public-key-retrieval-is-not-allowed/
database.allowPublicKeyRetrieval: "true"
snapshot.mode: Debezium can use different modes when it runs a snapshot. The snapshot mode is determined by the snapshot.mode configuration property.
snapshot.mode: "initial"
offset.flush.interval.ms: The number of milliseconds to wait before flushing recent offsets to Kafka. This ensures that offsets are committed within
the specified time interval. offset.flush.interval.ms: 5000
connector.class: The Java class for the connector. This must be set to io.debezium.connector.postgresql.PostgresConnector.
connector.class: "io.debezium.connector.postgresql.PostgresConnector"
offset.storage: The Java class that implements the offset storage strategy. This must be set to io.debezium.storage.jdbc.offset.JdbcOffsetBackingStor
e. offset.storage: "io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore"
offset.storage.jdbc.offset.table.name: The name of the database table where connector offsets are to be stored.
offset.storage.jdbc.offset.table.name: "altinity_sink_connector.replica_source_info"
offset.storage.jdbc.url: The JDBC URL for the database where connector offsets are to be stored.
offset.storage.jdbc.url: "jdbc:clickhouse://clickhouse:8123/altinity_sink_connector"
offset.storage.jdbc.user: The name of the database user to be used when connecting to the database where connector offsets are to be stored.
offset.storage.jdbc.user: "root"
offset.storage.jdbc.password: The password of the database user to be used when connecting to the database where connector offsets are to be stored.
offset.storage.jdbc.password: "root"
offset.storage.jdbc.offset.table.ddl: The DDL statement used to create the database table where connector offsets are to be stored.
offset.storage.jdbc.offset.table.ddl: "CREATE TABLE if not exists %s (
id
String,offset_key
String,offset_val
String,record_insert_ts
DateTime,record_insert_seq
UInt64,_version
UInt64 MATERIALIZED toUnixTimestamp64Nano(now64(9)) ) ENGINE = ReplacingMergeTree(_version) ORDER BY id SETTINGS index_granularity = 8198" offset.storage.jdbc.offset.table.delete: "delete from %s where 1=1" schema.history.internal: "io.debezium.storage.jdbc.history.JdbcSchemaHistory" schema.history.internal.jdbc.url: "jdbc:clickhouse://clickhouse:8123/altinity_sink_connector" schema.history.internal.jdbc.user: "root" schema.history.internal.jdbc.password: "root" schema.history.internal.jdbc.schema.history.table.ddl: "CREATE TABLE if not exists %s (id
VARCHAR(36) NOT NULL,history_data
VARCHAR(65000),history_data_seq
INTEGER,record_insert_ts
TIMESTAMP NOT NULL,record_insert_seq
INTEG ER NOT NULL) ENGINE=ReplacingMergeTree(record_insert_seq) order by id"schema.history.internal.schema.history.table.name: The name of the database table where connector schema history is to be stored.
schema.history.internal.jdbc.schema.history.table.name: "altinity_sink_connector.replicate_schema_history"
enable.snapshot.ddl: If set to true, the connector wil parse the DDL statements as part of initial load.
enable.snapshot.ddl: "true"
auto.create.tables: If set to true, the connector will create the database tables for the destination tables if they do not already exist.
auto.create.tables: "true"
database.dbname: The name of the PostgreSQL database from which events are to be captured when not using snapshot mode.
database.dbname: "demo"
clickhouse.datetime.timezone: This timezone will override the default timezone of ClickHouse server. Timezone columns will be set to this timezone.
clickhouse.datetime.timezone: "UTC"
skip_replica_start: If set to true, the connector will skip replication on startup. sink-connector-client start_replica will start replication.
skip_replica_start: "false"
binary.handling.mode: The mode for handling binary values. Possible values are bytes, base64, and decode. The default is bytes.
binary.handling.mode: "base64"
ignore_delete: If set to true, the connector will ignore delete events. The default is false.
ignore_delete: "true"
disable.ddl: If set to true, the connector will ignore DDL events. The default is false.
disable.ddl: "false"
disable.drop.truncate: If set to true, the connector will ignore drop and truncate events. The default is false.
disable.drop.truncate: "false"