[Bug] cdc action can not deal RENAME COLUMN correctly

[zhangjun0x01]

Search before asking

• [X] I searched in the issues and found nothing similar.

Paimon version

0.4

Compute Engine

flink

Minimal reproduce step

1. we have a mysql table ,like this:

mysql> select * from products;
+----+------+----------+-------+------+------+------------+---------------------+---------------------+
| ID | b    | CREATEBY | a     | c    | d    | e          | f                   | g                   |
+----+------+----------+-------+------+------+------------+---------------------+---------------------+
| 1  | aa   | aa       | 1.230 | bb   | ccc  | 2023-04-19 | 2023-04-19 12:43:11 | 2023-04-19 12:43:13 |
| 2  | aa   | bb       | 1.230 | bb   | ccc  | 2023-04-19 | 2023-04-19 12:43:11 | 2023-04-19 12:43:13 |
| 3  | aa   | c        | 2.300 | bb   | ccc  | 2023-04-19 | 2023-04-19 12:43:11 | 2023-04-19 12:43:13 |
+----+------+----------+-------+------+------+------------+---------------------+---------------------+

2. start the flink cdc action to sync the mysql data to paimon.
3. rename the mysql column a to a1
4. the paimon table will add a new column a1, and do not drop old column a.

Flink SQL> select * from products;
+----+----+----------+-------+----+-----+------------+---------------------+---------------------+--------+
| ID |  b | CREATEBY |     a |  c |   d |          e |                   f |                   g |     a1 |
+----+----+----------+-------+----+-----+------------+---------------------+---------------------+--------+
|  1 | aa |       aa | 1.230 | bb | ccc | 2023-04-19 | 2023-04-19 12:43:11 | 2023-04-19 12:43:13 | <NULL> |
|  2 | aa |       bb | 1.230 | bb | ccc | 2023-04-19 | 2023-04-19 12:43:11 | 2023-04-19 12:43:13 | <NULL> |
|  3 | aa |        c | 2.300 | bb | ccc | 2023-04-19 | 2023-04-19 12:43:11 | 2023-04-19 12:43:13 | <NULL> |
+----+----+----------+-------+----+-----+------------+---------------------+---------------------+--------+
3 rows in set

What doesn't meet your expectations?

The result does not meet expectations

Anything else?

No response

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[JingsongLi]

Thanks @zhangjun0x01 for reporting, this is by designed. The framework is hard to support true renaming. We should document this behavior.

[zhangjun0x01]

Thanks @zhangjun0x01 for reporting, this is by designed. The framework is hard to support true renaming. We should document this behavior.

yes ,it is difficult to distinguish ALTER TABLE t RENAME COLUMN a to a1 and ALTER TABLE t DROP a , ADD a1 int,

Although MySQL supports these two grammars, the change data obtained by debezium is the same (except for ddl)

[JingsongLi]

Thanks @zhangjun0x01 for reporting, this is by designed. The framework is hard to support true renaming. We should document this behavior.

yes ,it is difficult to distinguish ALTER TABLE t RENAME COLUMN a to a1 and ALTER TABLE t DROP a , ADD a1 int,

Although MySQL supports these two grammars, the change data obtained by debezium is the same (except for ddl)

Can you document this?

Currently, the framework can not drop columns, so the behaviors of `DROP ...` and `RENAME` are ... 

[zhangjun0x01]

Thanks @zhangjun0x01 for reporting, this is by designed. The framework is hard to support true renaming. We should document this behavior.

yes ,it is difficult to distinguish ALTER TABLE t RENAME COLUMN a to a1 and ALTER TABLE t DROP a , ADD a1 int, Although MySQL supports these two grammars, the change data obtained by debezium is the same (except for ddl)

Can you document this?

Currently, the framework can not drop columns, so the behaviors of `DROP ...` and `RENAME` are ... 

ok , I will do it later

[JingsongLi]

I found documentation is wrong: "are supported. Other type changes will cause exceptions.". https://paimon.apache.org/docs/master/how-to/cdc-ingestion/ Actually, there is no exception.