search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.46k stars 2.24k forks source link

Add view support for Hive catalog #8698

Closed ajantha-bhat closed 1 month ago

ajantha-bhat commented 1 year ago

similar to https://github.com/apache/iceberg/pull/7913/

ajantha-bhat commented 1 year ago

@nk1506: Would you like to work on this?

nk1506 commented 1 year ago

Yes, I would like to contribute on this. Thanks @ajantha-bhat

nk1506 commented 1 year ago

As part of this I have couple of questions ? For IcebergTable we use TableType as EXTERNAL_TABLE as well as properties with TABLE_TYPE_PROP.

With View Support:

  1. We can set TableType as VIRTUAL_VIEW
  2. It will make implementation more clean.
  3. We can also follow the strategy of managing extra field with table properties.
  4. To make backward compatibility we must add the required checks every where.
  5. IMetaStoreClient doesn't have specific APIs for Views.
  6. HiveTableOperations/HiveViewOperations must add all the required checks to manage same table/view name.

Please share your thoughts. CC: @ajantha-bhat , @jbonofre . Also tag the relevant people.

pvary commented 1 year ago

@deniskuzZ: Is there there something like this ongoing in the Hive codebase?

ajantha-bhat commented 1 year ago

Please share your thoughts. CC: @ajantha-bhat , @jbonofre . Also tag the relevant people.

I think we can use VIRTUAL_VIEW type to avoid adding custom properties to the table.

@pvary: Thanks for commenting. I was about to tag you. Since this is a change for HiveCatalog (to extend BaseViewOperations), I think it should be done in this repo? As per my knowledge, only the integration code with Hive 4.x is kept in Hive codebase.

pvary commented 1 year ago

@ajantha-bhat: The question is whether the Hive 3 HMS API is enough for the integration, or not. I would prefer if it would be enough, but we should definitely try to involve the Hive developers too.

deniskuzZ commented 1 year ago

@deniskuzZ: Is there there something like this ongoing in the Hive codebase?

hi @pvary, we have iceberg MV backed by iceberg table support in Hive where query definitions are stored in HMS and data in Iceberg table: HIVE-26578, HIVE-27101 Currently investigating how we could compute the delta between snapshots for incremental MV rebuild.

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

ajantha-bhat commented 1 month ago

Closed via https://github.com/apache/iceberg/pull/9852