Copy iceberg table from hdfs to GCS and register table to BLMS

apache / iceberg

Apache Iceberg

https://iceberg.apache.org/

Apache License 2.0

6.49k stars 2.24k forks source link

Copy iceberg table from hdfs to GCS and register table to BLMS #10389

Open Manoranjan93 opened 5 months ago

Manoranjan93 commented 5 months ago

Query engine

Dataproc

Question

Does iceberg table support in place migration - I have iceberg tables stored in hdfs location which have been copied to GCS location, Now I want to register those tables in big lake metastore (BLMS). Please suggest the best approach ?

Fokko commented 5 months ago

Hey @Manoranjan93 Thanks for reaching out!

Iceberg has absolute paths, so you cannot just copy the files as the paths in the metadata still point to S3. The best way to copy this is using a query engine such as Spark, Trino, etc.

Biglake is still underway, see https://github.com/apache/iceberg/pull/7412 I hope this helps!