apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.49k stars 2.24k forks source link

How does client use hadoopcatlog to read the iceberg table writen by hivecatalog? #11375

Open lookingUpAtTheSky opened 1 month ago

lookingUpAtTheSky commented 1 month ago

Query engine

spark

Question

In our scenario, we use hadoopcatalog to read the iceberg table stored on the oss. These tables are written through Hivecatalog, the table does not has version-hint.txt, so how can I read these tables using hadoopcatalog.

manuzhang commented 4 weeks ago

Then it will read the metadata file with max version.

RussellSpitzer commented 2 weeks ago

I strongly recommend not doing this, using different catalogs with the same table can produce inconsistent results. @manuzhang is correct though that theoretically it would be able to look for the highest prefixed metadata.json although this would not be guaranteed to be committed or even related to the table in the Hive metastore.