apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.38k stars 940 forks source link

[Bug] how to sync data from mysql to paimon #1560

Open homesickjava opened 1 year ago

homesickjava commented 1 year ago

Search before asking

Paimon version

05-SNAPSHOT

Compute Engine

flink

Minimal reproduce step

  1. run paimon-flink-action.jar
  2. use hive to query paimon table
  3. could not get data from paimon table

image

I have 8898 rows of data in source mysql table image

BUT there's no data sync into paimon table image

What doesn't meet your expectations?

I want to get sync data from source mysql table

Anything else?

No response

Are you willing to submit a PR?

EMsnap commented 1 year ago

I guess you need to provide more info on your job and make sure the configuration in mysqlcdc source and paimon sink are correct.

homesickjava commented 1 year ago

I have mysqlcdc depended jar in $FLINK_HOME/lib image

and I use flink to run paimon-flink-action.jar to synchronizing datas , and my command is : ./flink run -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.4-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/warehouse --database paimon_db --table paimon_sync_machine_baseinfo --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='machine_baseinfo' --catalog-conf metastore=hive --catalog-conf uri=thrift://10.206.32.3:9083 --table-conf bucket=4 --table-conf changelog-producer=input --table-conf sink.parallelism=4

I use both hive and flink sql-client to query paimon data of paimon_sync_machine_baseinfo . however, I got empty set of the quering, how can I synchronizing the data from mysql to paimon ~~

homesickjava commented 1 year ago

it has data synchronizing info, however, when I use flink-sql to query, I could not get any records the script that I use: ./flink run -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.4-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/warehouse --database paimon_db --table sync_machine_baseinfo_addcolumn --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='machine_baseinfo'

image