Tracking: Bummock as an indexing storage for various data models.

[wyhyhyhyh]

1. Relational Bummock will serve as a unified indexing interface for table storage built on top of Hummock. See https://singularity-data.quip.com/lYozAfUcsvXV/Bummock-as-an-indexing-storage for design doc.

A roadmap includes:

• [ ] Implement a unified interface for collection. https://github.com/singularity-data/risingwave-dev/pull/573
• [ ] Implement secondary index for table.
• [ ] Implement index scan batch executor.
• [ ] Implement index maintenance.

2. Nested (protobuf, json) We support nested data model an indexing interface. https://singularity-data.quip.com/63ipA5OTEqnD/Considerations-on-Schemaless-Bummock

A brief roadmap:

• [x] #591
• [x] #575
• [ ] Design and implement the storage format for nested data.
• [ ] Implement write path for nested data.
• [ ] Implement path scanner for nested data.

Long term:

• [ ] Implement indexing for nested data.

[neverchanje]

I think this issue mostly overlaps with https://github.com/singularity-data/risingwave-dev/issues/515 so maybe you should only include those that are not assigned here.

[wyhyhyhyh]

I think this issue mostly overlaps with #515 so maybe you should only include those that are not assigned here.

Yes, this tracking issue focuses on storage.

[fuyufjh]

We support protobuf data model as a separate storage engine in Bummock.

This description confuses me... From my understanding, the "Protobuf data model" is a data model and it must be supported in both Hummock and Bummock. In another word, "Protobuf data model" and "Bummock" are orthogonal topics for me... Is it correct?

[wyhyhyhyh]

We support protobuf data model as a separate storage engine in Bummock.

This description confuses me... From my understanding, the "Protobuf data model" is a data model and it must be supported in both Hummock and Bummock. In another word, "Protobuf data model" and "Bummock" are orthogonal topics for me... Is it correct?

In my understanding:

• Hummock is the KV layer at the bottom serving unified storage engine.
• Bummock is supposed to provide unified interface for collection access, e.g. scan, insert, delete etc. (Similar to the position of current ScannableTable, but an extension with more functionality and fine-grained interfaces). MViewTable, indexes, nested relation (so called protobuf data model) will all be wrapped up inside the Bummock.

Bummock is a framework, both relational model and protobuf data model (nested relation) fit inside.

[neverchanje]

There's no such thing as protobuf data model btw, even google doesn't name it when they invented dremel. The most widely used word is "nested data model".

[wyhyhyhyh]

There's no such thing as protobuf data model btw, even google doesn't name it when they invented dremel. The most widely used word is "nested data model".

True. Thanks.

[fuyufjh]

In my understanding:

• Hummock is the KV layer at the bottom serving unified storage engine.

• Bummock is supposed to provide unified interface for collection access, e.g. scan, insert, delete etc. (Similar to the position of current ScannableTable, but an extension with more functionality and fine-grained interfaces). MViewTable, indexes, nested relation (so called protobuf data model) will all be wrapped up inside the Bummock.

Bummock is a framework, both relational model and protobuf data model (nested relation) fit inside.

You mean Bummock will be the storage interface for compute layer instead of Hummock? That's the first time for me to heard that😥

[twocode]

You mean Bummock will be the storage interface for compute layer instead of Hummock? That's the first time for me to heard that😥

Interface between compute layer and storage is always Table (unchanged from the first day table was introduced). Bummock will provide indexing interfaces to support Table.

[wyhyhyhyh]

In my understanding:

• Hummock is the KV layer at the bottom serving unified storage engine.
• Bummock is supposed to provide unified interface for collection access, e.g. scan, insert, delete etc. (Similar to the position of current ScannableTable, but an extension with more functionality and fine-grained interfaces). MViewTable, indexes, nested relation (so called protobuf data model) will all be wrapped up inside the Bummock.

Bummock is a framework, both relational model and protobuf data model (nested relation) fit inside.

You mean Bummock will be the storage interface for compute layer instead of Hummock? That's the first time for me to heard that😥

Yes, very tricky and confusing for me also.

[wyhyhyhyh]

Closed as the bummock design needs revision.