opensearch-project / opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.
Apache License 2.0
20 stars 32 forks source link

OpenSearch index naming logic for create MV #562

Open noCharger opened 2 months ago

noCharger commented 2 months ago

For glue type data source, Current create MV logic doesn't validate the existence of Database nor Table of glue data catalog.

CREATE MATERIALIZED VIEW dummyDatasource.dummyDB.dummyTable.dummyMV ****
CREATE MATERIALIZED VIEW dummyDB.dummyTable.dummyMV ****

Both will create OpenSearch index with nameflint_dummyDatasource_dummyDB_dummyTable_dummyMV

CREATE MATERIALIZED VIEW dummyMV ****

This will create OpenSearch index with nameflint_dummyDatasource_default_dummyMV

CREATE MATERIALIZED VIEW dummyDatasource.dummyMV ****

This will create OpenSearch index with nameflint_dummyDatasource__dummyMV

CREATE MATERIALIZED VIEW dummyTable.dummyMV ****
CREATE MATERIALIZED VIEW dummyDatasource.dummyTable.dummyMV ****

Both will create OpenSearch index with name flint_dummyDatasource_dummyTable_dummyMV

CREATE MATERIALIZED VIEW dummyDB.dummyMV ****
CREATE MATERIALIZED VIEW dummyDatasource.dummyDB.dummyMV ****

Both will create OpenSearch index with name flint_dummyDatasource_dummyDB_dummyMV

Open to discuss: What's the user experience can be consolidated? Should the glue resources being validated before actually create the flint index?

noCharger commented 2 months ago

One option to mitigate this issue: Add index name to FlintMetadataLogEntry, to decouple the index naming logic from recover index itself.

dai-chen commented 1 month ago

Related: https://github.com/opensearch-project/sql/issues/2908