[RFC] OpenSearch Data Format

[penghuo]

User Pain point

• Not Open Format Dataset: Currently, all the data indexed in the OpenSearch service. Users rely on the OpenSearch service to access and retrieve the dataset.
• Lacking segregation of read and write operations : using single cluster serve Index traffic and query traffic.
• Complex data management workflow: usually, user need to setup pipeline to (1) ingest data to S3, and then (2) ingest to OpenSearch and (3) then manage index lifecycle with customized rules.

Proposed Solution

• The index themselves are encoded with Lucene format of each shard.
• The metadata is in object store. The metadata also include skipping index such as mix/max for each shard.
• No server need to be running on to maintain OpenSearch Index. Transactions are achieve using optimistic concurrency protocol.
• User only need to launch server when run queries, and benefits of separately scaling compute and storage.

Technical Challenge

• OpenSearch Data Format on Object Store
  • OpenSearch Data Format structure on object store
  • Metadata specification. Using transaction log to record actions. The actions include (1) Add/Remove (2) Metadata Change
  • Access Protocols
  • Optimistic concurrency protocols
  • Serializable isolation
• Implement OpenSearch Data Format Writer/Reader as library
• Implement Virtual Index in OpenSearch which attached to OpenSearch Data Format
• Implement DSL query rewrite logic with skipping index.

[penghuo]

Demo setup

• Demo Streaming application
  • Write 1 shard to fs every 5s
  • Write skipping index for each shard.
• Reuse searchable snapshot interface
  • restore unassigned shard from fs every 10s.
• Rewrite DSL query WITH skipping index

[dai-chen]

Here is the demo video that covers the following topic:

1. OpenSearch Data Format proposed in this issue that remove hard dependency on OpenSearch cluster and separate read and write path
2. Virtual / External Index that makes data set on object store accessible to OpenSearch. Please find more details in https://github.com/opensearch-project/sql/issues/1080
3. Skipping Index that avoids unnecessary shard load and scan. Please find more details in https://github.com/opensearch-project/opensearch-spark/issues/2

https://github.com/opensearch-project/OpenSearch/assets/46505291/b2b71f27-3f55-4f31-815c-ede2df6e5aa2

[schenksj]

this is very cool! has any progress been made on the spark sql execution datasources side?