ExpediaGroup / waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.
Apache License 2.0
273 stars 76 forks source link

enable READ_WRITE support for federated metastores #87

Closed rpoluri closed 6 years ago

rpoluri commented 6 years ago

As a user of WD I like to run DDL on federated metastores So that I can run ETL againt primary metastore and update federated central datalake.

Acceptance Criteria: ability to configure access-control-type option for federated metastores with atleast following possible values READ_ONLY, READ_WRITE.

teabot commented 6 years ago

Some additional information. There are a number of write operations in the metastore whose requests do not contain database/table name context, and so cannot be routed to federated instances. We believe that this is not an issue for general operation, but may be a problem if you are wanting to use certain specific Hive features. At this time we believe the following could not be supported in a writable federation model:

patduin commented 6 years ago

You can add create database to that list

rpoluri commented 6 years ago

Thanks for the info, our current use-case is limited to managing tables and partitions through waggle-dance federation, so should be fine with above limitations.

I also think anything that is not specific to schema should be managed by datalake owner.