apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.39k stars 942 forks source link

[Bug] MysqlCdcAction can not convert decimal type correct #943

Closed zhangjun0x01 closed 1 year ago

zhangjun0x01 commented 1 year ago

Search before asking

Paimon version

0.4

Compute Engine

flink

Minimal reproduce step

  1. start a flink cdc action , sync the mysql data to paimon. the mysql table has a column with decimal type.
    
    mysql> SELECT * FROM products;
    +----+----------+-------+
    | ID | CREATEBY | a     |
    +----+----------+-------+
    | 1  | aa       | 1.230 |
    | 2  | bb       | 1.230 |
    +----+----------+-------+
    2 rows in set (0.00 sec)

mysql> desc products; +----------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +----------+---------------+------+-----+---------+-------+ | ID | varchar(100) | NO | PRI | NULL | | | CREATEBY | varchar(100) | YES | | NULL | | | a | decimal(10,3) | YES | | NULL | | +----------+---------------+------+-----+---------+-------+ 3 rows in set (0.01 sec)



2. stop the flink job
3. start the flink job again, the paimon will throw an exception.

### What doesn't meet your expectations?

java.lang.IllegalArgumentException: Paimon schema and MySQL schema are not compatible.
Paimon fields are: [`ID` VARCHAR(100) NOT NULL, `CREATEBY` VARCHAR(100), `a` DECIMAL(10, 3)].
MySQL fields are: {ID=VARCHAR(100), CREATEBY=VARCHAR(100), a=DECIMAL(10, 3)}

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!
JingsongLi commented 1 year ago

longtext also not work?

zhangjun0x01 commented 1 year ago

longtext also not work?

I'll check the other types again

zhangjun0x01 commented 1 year ago

longtext also not work?

I check it ,found that longtext throws an exception Don't support MySQL type ,I add the longtext and longblob in MySqlTypeUtils#toDataType.

time datatype is also don't support, but I think it may be complex, It will involve issues of timezone and precision,I will fix it in another pr。

I also add the date type to check compatibility.

JingsongLi commented 1 year ago

Thanks @zhangjun0x01