Closed imotai closed 1 year ago
[propose the document level ownership #271](https://github.com/dbpunk-labs/db3/issues/271)
Database → Collection → Document
flowchart LR
A[Developers] --> B[database]
B --> C[colection 1]
C --> E[document 1]
C --> F[document 2]
C --> G[document 3]
B --> D[colection 2]
D --> H[document 4]
D --> I[document 5]
J[User] --> |"(Document, Signature, PublicKey)"|D
ID Type | Size | Description |
---|---|---|
DbId | 20 | DbId := DB3Address |
TxId | 32 | transaction id |
AccountId | 20 | AccountId := DB3Address |
BlockID | 8 | Current block height |
MutationID | 4 | Index of mutation in the current block. |
PutEntryIdx | 4 | Index of document in current database mutation |
PutEntryID | 16 | PutEntryID := BlockID + MutationID + PutEntryIdx |
CollectionID | 16 | CollectionID := PutEntryID |
DocumentEntryID | 16 | DocumentEntryID := PutEntryID |
DocumentID | 33 | DocumenID := DocumentEntryType+CollectionID + DocumentKeyID |
IndexFiledID | 4 | Index of the index filed in the collection |
IndexID | KeyBytes + 54 | IndexId := IndexEntryType+CollectionID + IndexFiledID + KeyBytes + DocumentID |
https://github.com/dbpunk-labs/db3/pull/297
sequenceDiagram
Client->>+Node: 1.1 CreateDocument(document)
Node->>+DocumentImpl:1.3 SubmitMutation(WRITE_DOCUMENT, document)
Node-->>-Client: 1.4 Response(status, msg)
DocumentImpl->>DocumentImpl: 2.1 documentId = GenerateDocumentId(blockId, autoId)
DocumentImpl->>+KVStore: 2.2 InsertKV(documentId, txid, document)
KVStore-->>-DocumentImpl: 2.3 Response(status, msg)
DocumentImpl-->>DocumentImpl: 2.4 index_key_pairs = GetIndexKeyPairs(db, collection, document)
loop Create Indexes: index_key_pairs.foreach
DocumentImpl-->>DocumentImpl: 2.5.1 GenerateIndexId(db, collection, documentId, index, key)
DocumentImpl->>+KVStore: 2.5.2 InsertKV(indexId, txid, documentId)
KVStore-->>-DocumentImpl: 2.5.3 Response(status, msg)
end
DocumentImpl-->>-Node: 2.6 Response(status, msg, documentId)
sequenceDiagram
Client->>+Node: 1.1 UpdateDocument(documentId, document)
Node->>+DocumentImpl:1.2 SubmitMutation(UpdateDocument, documentId, document)
Node-->>-Client: 1.3 Response(status, msg)
DocumentImpl->>+KVStore: 2.1 GetKV(documentId)
KVStore-->>-DocumentImpl: 2.2 Response(status, msg, address, old_document)
DocumentImpl-->>DocumentImpl: 2.3 CheckOwnership(Signature, PublicKey, address)
DocumentImpl-->>DocumentImpl: 2.4 modified_index_key_pairs = GetModifiedIndexKeyPairs(db, collection, old_document, document)
DocumentImpl->>+KVStore: 2.5 Update documet: InsertKV(documentId, txid, document)
KVStore-->>-DocumentImpl: 2.6 Response(status, msg)
DocumentImpl->>DocumentImpl: 2.7 (indexes, keys) = CheckModifiedIndexs(document, index_keys)
loop Update Indexes: modified_index_key_pairs.foreach(index, key)
DocumentImpl-->>DocumentImpl: 2.8.1 indexId = GenerateIndexId(db, collection, index, key, documentId)
DocumentImpl->>+KVStore: 2.8.2 InsertKV(indexId, txid, documentId)
KVStore-->>-DocumentImpl: 2.8.3 Response(status, msg)
end
DocumentImpl-->>-Node: 2.9 Response(status, msg)
sequenceDiagram
Client->>+Node: 1.1 DeleteDocument(documentId)
Node->>+DocumentImpl:1.2 SubmitMutation(DeleteDocument, documentId)
Node-->>-Client: 1.3 Response(status, msg)
DocumentImpl->>+KVStore: 2.1 GetKV(documentId)
KVStore-->>-DocumentImpl: 2.2 Response(status, msg, index_keys, txid)
DocumentImpl-->>DocumentImpl: 2.3 CheckOwnership(Signature, PublicKey, txid)
DocumentImpl->>+KVStore: 2.4 Delete documet: DeleteKV(documentId)
KVStore-->>-DocumentImpl: 2.5 Response(status, msg)
DocumentImpl-->>DocumentImpl: 2.6 index_key_pairs = GetIndexKeyPairs(db, collection, document)
loop Delete Indexes: index_key_pairs.foreach(index, key)
DocumentImpl-->>DocumentImpl: 2.7.1 GenerateIndexId(db, collection, index, key, documentId)
DocumentImpl->>+KVStore: 2.7.2 DeleteKV(indexId, txid)
KVStore-->>-DocumentImpl: 2.7.3 Response(status, msg)
end
DocumentImpl-->>-Node: 2.8 Response(status, msg)
sequenceDiagram
Client->>+Node: 1.1 GetDocumentById(documentId)
Node->>+DocumentImpl:1.3 GetDocumentById(documentId)
DocumentImpl->>+KVStore: 2.1 BatchGetKey(documentId)
KVStore-->>-DocumentImpl: 2.2 Response(status, msg, document)
DocumentImpl-->>-Node: 2.5 Response(status, msg)
Node-->>-Client: 1.4 Response(status, msg)
Client->>+Node: 2.1 GetDocuments(index, key)
Node->>+DocumentImpl:2.2 GetDocuments(index, key)
DocumentImpl->>DocumentImpl: 2.3 index_range = GenerateIndexRange(db, collection, index, key)
DocumentImpl->>+KVStore: 2.3 GetRange(index_range)
KVStore-->>-DocumentImpl: 2.4 Response(status, msg, documents)
DocumentImpl-->>-Node: 2.5 Response(status, msg, documents)
Node-->>-Client: 2.6 Response(status, msg, documents)
@imotai
Regarding to the rpc proto definition mentioned here, one document might be under multiple collections? Will we define and implement collection in the same way?
I find those concepts are used as defined in the google firestore.
Discussion
@imotai
1. What is the relationship between collection and documents? 1:N or N:M?
Regarding to the rpc proto definition mentioned here, one document might be under multiple collections? Will we define and implement collection in the same way?
2. Will we support Project, Namespace, Collection Group, in db3.
I find those concepts are used as defined in the google firestore.
DB3 will design and implement a set of APIs which are similar to google firestore so that the developers who are familiar to firestore can transfer to DB3 easily.
This API provides several administrative services for DB3 store.
API | Description | |
---|---|---|
CreateDatabase | Create database and linked with address | |
CreateCollection | Define indexes | |
ListIndexes | Lists composite indexes. | |
GetIndex | Gets a composite index. | |
GetDatabase | Gets information about a database. | |
GetField | Gets the metadata and configuration for a Field. | |
ListFields | Lists the field configuration and metadata for this database. |
API | Description | |
---|---|---|
UpdateDatabase | Updates a database. | |
UpdateField | Updates a field configuration. Currently, field updates apply only to single field index configuration. | |
ExportDocuments | Exports a copy of all or a subset of documents from DB3 store | |
ImportDocuments | Imports documents into DB3 store. Existing documents with the same name are overwritten. | |
ListDatabases | List all the databases in the project. | |
CreateIndex | Creates a composite index. This returns a db.longrunninbg.Operation (similar to google.longrunning.Operation) which may be used to track the status of the creation. The metadata for the operation will be the type IndexOperationMetadata | |
DeleteIndex | Deletes a composite index. |
API | Description | |
---|---|---|
CreateDocument | Creates a new document. | |
GetDocument | Gets a single document. | |
ListDocuments | List documents | |
UpdateDocument | Updates or inserts a document. | |
DeleteDocument | Delete a document |
API | Description | |
---|---|---|
BatchGetDocuments | Gets multiple documents. Documents returned by this method are not guaranteed to be returned in the same order that they were requested. | |
ListCollectionIds | List documents | |
RunQuery | Runs a query. | |
RunAggregationQuery | Return the number of documents in table given a filter. SELECT COUNT(*) FROM ( SELECT * FROM k where a = true ); |
|
PartitionQuery | Partitions a query by returning partition cursors that can be used to run the query in parallel. |
Entity | Description | |
---|---|---|
Database | Gets multiple documents. Documents returned by this method are not guaranteed to be returned in the same order that they were requested. | |
Field | Represents a single field in the database. Fields are grouped by their "Collection Group", which represent all collections in the database with the same id. | |
Index | indexes enable simple and complex queries against documents in a database. | |
Document | A db3 document. Must not exceed 1 MiB - 4 bytes. |
Milestore | Task | Owner | Status |
---|---|---|---|
Design POC | Document Operation workflow design | @cj | Completed |
Design POC | DB3 admin opertation workflow design | @wtz | Completed |
Feature Implementation | Entity Proto implement P0 | @wtz | In Progress |
Feature Implementation | DB3 Admin P0 Op Implementation | @wtz | In Progress |
Feature Implementation | DB document P0 Op implementation | @cj | Not started |
QA | Unit test & integartion test | @cj | Not started |
QA | Benchmark | @cj | Not started |
JS SDK API | Implement JS SDK for db3 admin and document operation | @zhaojun | Not started |
>,<,>=,<=
filter condition fro run query document with index #404 @imotai 🔔 👀 ♨️ I wanna introduce CollectionID (16 bytes) as a part of IndexID instead of using colecction_name varchar().
IndexId = |DbID(20)|CollectionId(16)|BlockID(16)|
Discussion
@imotai 🔔 👀 ♨️ I wanna introduce CollectionID (16 bytes) as a part of IndexID instead of using colecction_name varchar().
IndexId = |DbID(20)|CollectionId(16)|BlockID(16)|
agree
Further work will be taken care by https://github.com/dbpunk-labs/db3/issues/321
Motivations
Solution