apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.71k stars 1.94k forks source link

The main method caused an error: Invalid metadata key 'row_kind' in column 'operation' of table #3117

Closed KONEONE closed 8 months ago

KONEONE commented 8 months ago

我正在运行如下sql:


CREATE TABLE IF NOT EXISTS test_table (
    db_name       STRING METADATA FROM 'database_name' VIRTUAL,
    table_name    STRING METADATA FROM 'table_name' VIRTUAL,
    operation_ts  TIMESTAMP_LTZ(3) METADATA FROM 'op_ts' VIRTUAL,
    operation     STRING METADATA FROM 'row_kind' VIRTUAL,
    id BIGINT NOT NULL COMMENT '编号',
    PRIMARY KEY ( id ) NOT ENFORCED
) WITH (
    'connector' = 'mysql-cdc',
    'hostname' = 'localhost',
    'port' = '3306',
    'username' = 'root',
    'password' = '123456',
    'database-name' = 'mydb',
    'table-name' = 'test'
);

image 在上面已经存在mysql-cdc:3.0.1了, 想要去获取row_kind字段,但是报错:

Caused by: org.apache.flink.client.deployment.application.ApplicationExecutionException: Could not execute application.
    ... 13 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Invalid metadata key 'row_kind' in column 'operation' of table 'default_catalog.default_database.gmall_cart_info'. The DynamicTableSource class 'com.ververica.cdc.connectors.mysql.table.MySqlTableSource' supports the following metadata keys for reading:
table_name
database_name
op_ts
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist-1.16.3.jar:1.16.3]
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.16.3.jar:1.16.3]
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:98) ~[flink-dist-1.16.3.jar:1.16.3]
    at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) ~[flink-dist-1.16.3.jar:1.16.3]
    ... 12 more
Caused by: org.apache.flink.table.api.ValidationException: Invalid metadata key 'row_kind' in column 'operation' of table 'default_catalog.default_database.gmall_cart_info'. The DynamicTableSource class 'com.ververica.cdc.connectors.mysql.table.MySqlTableSource' supports the following metadata keys for reading:
table_name
database_name
op_ts

难道是需要如下版本才可以吗?

<dependency>
  <groupId>com.ververica</groupId>
  <artifactId>flink-connector-mysql-cdc</artifactId>
  <!-- 请使用已发布的版本依赖,snapshot版本的依赖需要本地自行编译。 -->
  <version>3.0-SNAPSHOT</version>
</dependency>
KONEONE commented 8 months ago

可以给出2.4版本如何获取row_kind字段吗?

VincentYuuu commented 8 months ago

I use the flink-connector-mysql-cdc version 3.0.1, ran into the same issue. Can anyone check this please?

KONEONE commented 8 months ago

I use the flink-connector-mysql-cdc version 3.0.1, ran into the same issue. Can anyone check this please?

I once reported this issue in the official group chat, but did not get any feedback.

PatrickRen commented 8 months ago

Considering collaboration with developers around the world, please re-create your issue in English on Apache Jira under project Flink with component tag Flink CDC. Thank you!

duanjianSnail commented 1 month ago

You mentioned that instead of flink-connector-mysql-cdc, one should use flink-sql-connector-mysql-cdc. And i confirmed that flink-sql-connector-mysql-cdc-3.1.1 includes the row_kind feature, and also suggested trying flink-sql-connector-mysql-cdc-3.0.1