pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
428 stars 286 forks source link

Verify whether all MQ output protocol handles time zones correctly #1639

Open liuzix opened 3 years ago

liuzix commented 3 years ago

Recently it has come to our attention that our Avro output does not handle correctly the situation where the --tz option used in TiCDC is different from the local time zone. We need to verify whether all our protocols can handle time zones correctly, and, if not, under what condition things will go wrong and whether there is any workaround.

Checklist

Rustin170506 commented 3 years ago

/assign

I am working on the Canal JSON.

Rustin170506 commented 3 years ago

The Canal JSON protocol should not have this problem and is tested as follows:

  1. start a tidb cluster
  2. start a ticdc cluster
  3. create a changefeed with canal json protocol
  4. create an table:
    
    create table if not exists test.table_name
    (
    column_1 int not null,
    column_2 datetime null,
    column_3 timestamp null,
    constraint table_name_column_1_uindex
        unique (column_1)
    );

alter table test.table_name add primary key (column_1);

5. insert `23,2021-11-01 17:43:24,2021-11-01 17:43:33`
6. the kafka puput:
```json
{"id":0,"database":"test","table":"table_name","pkNames":["column_1"],"isDdl":false,"type":"INSERT","es":1635759848635,"ts":0,"sql":"","sqlType":{"column_1":-5,"column_2":93,"column_3":93},"mysqlType":{"column_1":"int","column_2":"datetime","column_3":"timestamp"},"data":[{"column_1":"23","column_2":"2021-11-01 17:43:24","column_3":"2021-11-01 17:43:33"}],"old":[null]}

Technically we are generating the string directly, so if the user's upstream and kafka consumer programs handle the time zone correctly (keeping it consistent), then there is no problem.

asddongmen commented 9 months ago

@3AceShowHand Could you please take a look at this issue?