Is your feature request related to a problem?
Flint core assumes storage for FlintMetadataLog to be OpenSearch. This meta issue aims to refactor code components and data model that are coupled with OpenSearch, and allow plugging in custom implementations using different storage.
What solution would you like?
377
Refactor OpenSearch IRestHighLevelClient creation with static factory method
Remove FlintOpenSearchMetadataLog dependency on FlintClient using the static factory
378
379
Extract metadata log operations from FlintClient into FlintMetadataLogService
FlintClient.getIndexMetadata no longer attaches metadata log entry. Instead, FlintSpark.describeIndex does this
Abstract FlintSparkIndexMonitor record heartbeat action
389
Move interfaces to flint-commons module
Use reflection for FlintMetadataLogServiceBuilder
Add parameter for building FlintMetadataLogService reading from spark properties
PR
Update FlintMetadataLogEntry and DefaultOptimisticTransaction to be generic to data storage
Remove dataSourceName from DefaultOptimisticTransaction
Issue? do we need OpenSearch index name sanitize for metadata log index
Is your feature request related to a problem? Flint core assumes storage for
FlintMetadataLogto be OpenSearch. This meta issue aims to refactor code components and data model that are coupled with OpenSearch, and allow plugging in custom implementations using different storage.What solution would you like?
377
IRestHighLevelClientcreation with static factory methodFlintOpenSearchMetadataLogdependency onFlintClientusing the static factory378
379
FlintClientintoFlintMetadataLogServiceFlintClient.getIndexMetadatano longer attaches metadata log entry. Instead,FlintSpark.describeIndexdoes thisFlintSparkIndexMonitorrecord heartbeat action389
FlintMetadataLogServiceBuilderFlintMetadataLogServicereading from spark propertiesFlintMetadataLogEntryandDefaultOptimisticTransactionto be generic to data storageDefaultOptimisticTransactionWhat alternatives have you considered? N/A
Do you have any additional context? N/A