opensearch-project / opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.
Apache License 2.0
12 stars 18 forks source link

Support custom metadata log service implementation #389

Closed seankao-az closed 5 days ago

seankao-az commented 1 week ago

Description

  1. Use reflection for FlintMetadataLogServiceBuilder to support custom implementation of FlintMetadataLogService
  2. Read SparkConf for such builder
  3. Make FlintMetadataLogService an abstract class with a constructor using SparkConf
  4. Move interfaces to flint-commons
    1. Moved files: FlintMetadataLog, FlintMetadataLogEntry, FlintMetadataLogService, OptimisticTransaction
    2. Remove unused failLogEntry from FlintMetadataLogEntry (no access to org.opensearch.index namespace). Can add back in next PR when storage is abstracted away from the common FlintMetadataLogEntry

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.