📜 Proposal for WDB-Archipelago - v2.0

[TanmoySG]

WDB-Archipelago

WunderDB v2.0, named "Archipelago", aims at making wdb code more maintainable and mature. This Proposal Issue will act as a place to put ideas for the v2.0. The main goal of v2.0 proposal should primarily encompass, but not limited to

• [x] Code Maturity

  • Composite and Facade based code.
  • Bigger components broken down into smaller chunks
  • Reusability of Code. Removal of Similar code with Generic ones.

• [x] Usage Maturity

  • Using Standard Usage Methods, Interaction Methods like HTTP Verbs etc
  • Mature Schema Usage - using JSONSchema

• [x] Granular Storage

  • Should use granular and non-blocking storage for data
  • Data should be available readily
  • Can use Graceful Shutdown to put data in file and keep data in memory otherwise

[TanmoySG]

https://stackoverflow.com/questions/25896322/how-can-i-create-a-separate-namespace-within-a-python-script

[TanmoySG]

Development Maturity

• Replace the "Unified Actions Endpoint/API" with entity (cluster, DB, collection, etc) differentiated API endpoints and Actions Routes - v2/{entity}/{action} . Example
  • Cluster EPs can be v2/cluster/create, v2/cluster/delete.
  • Similarly for data, EPs can be v2/data/add, v2/data/delete etc.
• EPs can also be differentiated Endpoints and action routes can replaced with HTTP Verbs. Like
  • GET for Read/Get Data (or any other entity value)
  • POST for adding Data (or any other entity value)
  • PATCH for updating Data (or any other entity value)
  • DELETE for deleting Data (or any other entity value)
  • Reference for HTTP Verbs
  • Example - POST v2/data/ to add data , PATCH v2/data/ to update data, and more.

[TanmoySG]

Code Modularity

• Break and Modularize Existing Code into Smaller chunks.
• Club similar or of same "origin" into same "packages". Eg. Code for cluster actions should be clubbed in a "clusters" package.
• Packages should be separated into files for modularity.
• Common Reusable code - "helper" code should be put in separate packages accessible to any components.
• Utility Functions/Code should also be part of helpers

[TanmoySG]

Code Maturity

• Use Design Patterns that hide implementation and can facilitate usage without worrying much about the underlying flow.
• One such design pattern is - Composite Design Pattern that can help in designing independent code blocks that can have zero (or little to none) dependancy on other components. These smaller chunks then come together in a "flow" performing their own actions and forming the end-to-end flow for a broader task, instead of a monolithic single code block to perform every task itself.
• The Facade design pattern can be used to hide implementation of any task and only focus on taking a set of arguments and processing it and providing the output without the user worrying about the implementation internally.
• Code Reusability is another factor to be taken care of.

[TanmoySG]

Schema

Currently the schemas are defined as fieldName : description and doesn't handle data-type, constraints. In v2,

• Option for using Schema-less and schema-full Collections will be allowed.
• Schema-Less Collections will have an open format, no/low constraints data storage, providing users flexibility when required. Users will be able to put/store any non-structured or non-schema data.
• Schema-Full Collections will have Strict Schema Enforcing with schema defined using JSONSchema. Each Data block will have to be complaint with the schema and must strictly follow the schema defined. Any data that isn't in accordance to the JSONSchema will be considered "illegal" and will not be inserted into the Collection.
• Schema-Full Collections will be able to use any one of the fields as primary key or identifier instead of the _id auto-generated ID field used as identifier in v1
• Schema-less Collections will still have the _id as primary identifier.
• JSONSchema - https://json-schema.org/
• Also can use a "fork" of JSONSchema - https://github.com/TanmoySG/w-service-manager/blob/service-onboarding/schema/schema.template.json
• Tutorial for Schema https://www.mongodb.com/basics/json-schema-examples#:~:text=JSON%20Schema%20is%20an%20IETF,validity%20across%20similar%20JSON%20data.
• Schema Definition will be JSON Doc based.
• (From a UI PoV) Schema Definition will no longer be per field basis, and will have a Single Text Box to put JSON Schema in.
• As additional UI feature (feature extension) UI can provide a way to generate JSON Schema from Example JSON Object and a Schema Builder UI to create JSON Schema using text boxes (field name), drop-downs (data types) and other UI elements for other Schema Definition Utilities (constraints, limit, etc)
• Can also use JSON Table Schema (JTS) http://dataprotocols.org/json-table-schema/

[TanmoySG]

Data

• Patching of Data (Data Updates) will be done primarily through JSONPatch.
• The marker will stay but will now have a different structure to define. Instead of marker : "key : value" that fails if there is even one extra gap before or after the :, the new Marker definition will use marker : {Key : <field-name> , Value : <Value of Field>}
• Bulk Data CRUD - Create, Read, Update and Delete Bulk Data in wDB-2.0 in a single request. Helpful for aggregated data transfer.

[TanmoySG]

Storage

Sample Directory Structure

[TanmoySG]

File System

• L1

• L2

• L3

• L4

• L5

• L6

• L7

[TanmoySG]

Notes

• Active Data can be stored in-memory, in-program, while running, instead of constant Read/Writes to file to reduce complexities and improve performance.
• Every Operations, on Data, Collections, Databases, Namespace, etc., can be first executed on the in-memory data and the operations can be put in a "Task Queue" in parallel that executes the operations on the data in the file, in an async manner, in a separate thread/process that runs in parallel to the actual actions on the in-memory data.
• Two (or more) Queues for executing tasks (operations) can be created. One primary Queue (with more resources and priority and performance oriented) for in-memory data operations and other (with lesser priority than primary q) for the extra tasks - including file operations.
• Another way of dealing with this is, at particular intervals, the active in-memory data is synced to the file in the same Task Queue Fashion just in-place of individual tasks, entire data is put in the queue.
• Graceful Shut-down can involving moving data into files with priority, creating and utilizing multiple (including primary) queue to put data in files in parallel and fast.

[TanmoySG]

Notes

• Light Query Language
  • DEFINE namespace Namespace-Name
  • DEFINE database db-name
  • etc
• Common Access Endpoint for a single access endpoint.
  • api/cluster/create or POST api/cluster maps to api/common with payload of either the Query or JSON Object with specification of the action.
• Standard Response and Error Codes
  • Use a JSON structure for standard errors
  • Can contain - ERROR_CODE, RESPONSE_STATUS_CODE (HTTP Status Code), RESPONSE_CODE (interchangeable with ERROR_CODE when error occurs), ERRROR/RESPONSE_MESSAGE, RESPONSE_TYPE (error or message)

[TanmoySG]

Notes

• Initial v2 will only use Namespace Level ACL

[TanmoySG]

Notes

In program Data Structure (with sample data JSON), derived from this Directory tree.

{
    "namespaces": {
        "gns": {
            "alias": "",
            "id": "",
            "path": "",
            "databases": {
                "db-one": {
                    "name": "db1",
                    "collection": {
                        "coll-one": {
                            "name": "",
                            "path": "",
                            "data": {
                                "data": {
                                    "field-1": "true"
                                },
                                "path": ""
                            },
                            "metadata": {
                                "metadata": null,
                                "schema": null,
                                "path": ""
                            }
                        }
                    },
                    "metadata": {
                        "metadata": null,
                        "schema": null,
                        "path": ""
                    },
                    "path": ""
                }
            }
        }
    },
    "users": {}
}

Go Model on which this is based

package model

type FilePath string
type Identifier string

type User map[string]interface{}

type WDBFileSystem struct {
    Namespaces map[Identifier]Namespace `json:"namespaces"`
    Users      map[Identifier]User      `json:"users"`
}

type Namespace struct {
    NamespaceAlias string                  `json:"alias"`
    NamespaceID    Identifier              `json:"id"`
    NamespacePath  FilePath                `json:"path"`
    Databases      map[Identifier]Database `json:"databases"`
}

type Database struct {
    DatabaseName Identifier                `json:"name"`
    Collections  map[Identifier]Collection `json:"collection"`
    Metadata     Metadata                  `json:"metadata"`
    DatabasePath FilePath                  `json:"path"`
}

type Collection struct {
    CollectionName Identifier `json:"name"`
    CollectionPath FilePath   `json:"path"`
    Data           Datum      `json:"data"`
    Metadata       Metadata   `json:"metadata"`
}

type Datum struct {
    Data     map[Identifier]interface{} `json:"data"`
    DataPath FilePath                   `json:"path"`
}

type Metadata struct {
    Metadata     map[string]interface{} `json:"metadata"`
    Schema       map[string]interface{} `json:"schema"`
    MetadataPath FilePath               `json:"path"`
}

[TanmoySG]

Storing JSON Files to Persist Data

A minified JSON File is stored JSON Data without indentation. This results in a lesser sized data file. Here is a comparison.

So the release build of WDB should use minified JSON to persist data after shutdown.

[TanmoySG]

Notes

• For go, export the "operator" functions that does the flow for each actions, and make it possible for go devs to use wdb repo as a library to embed into go programs.
• In a future release make wdb embeddable for go code so that it can be used without running the server and embed it.
• /healthz endpoint for exposing performance stats.

[TanmoySG]

Filters (replacement for Marker)

type Filter struct {
    Key   string      `json:"key"`
    Value interface{} `json:"value"`
    // Filter operator - < / greater , > / lesser , = / equal , != / unequal
    // Operator string      `json:"operator"` // Future scope
}

[TanmoySG]

https://mrkaran.dev/posts/barreldb/