[spark] Migrate should guard for file format option which may not accurate to the hive table

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.16k stars 855 forks source link

Closed xuzifu666 closed 1 month ago

xuzifu666 commented 1 month ago

create a hive table using parquet format named test.hive_table;
insert into hive table with several records;
CALL sys.migrate_table(source_type =>'hive', table => 'test.hive_table', options => 'file.format=orc');
insert into test.hive_table;
data file format would not all with parquet;