[DRAFT] Library Enrichment Proposal (codename: Nekdis)

[Didas-git]

The sole purpose of this pull request is to show this proposal to more people and gather community feedback about it. Me and @guyroyse already had some discussions about the proposal and i am aware that there is a lot that can be improved/changed and that even then the proposal might not be fully implemented, but the most important thing is to find what the developers and community that use redis-om want and thats why we want to know what features you guys like the most and would like to see in redis-om and which ones you don't like, we want to ear your feedback about both the current redis-om@beta and the proposal.

This proposal includes some new key features like, tuples, other rarray types, references, default values, required fields, custom methods and much more.

All the information about this proposal can be found in its repository.

Remember the proposal is in beta state so there will be bugs and if you find one feel free to report it

Feel free to contact any of us here or on the redis discord.

Fixes:

25 #28 #44 #54 #55 #66 #69 #120 #141 #184

[Didas-git]

As of Nekdis v0.8 Vector Similarity Search was added, would be great to get some feedback specially on the syntax used.

This fixes #85 and is directly related to #104

[Didas-git]

Nekdis v0.10 brings some desired changes like reduce clutter saved and better api for modules, the biggest change however is the change from required to optional meaning that all the fields are required by default.

I will be working on bringing all the files to this pull request during this week but i would like to point people who are reading this PR to the issues i created on the Nekdis repository since they are really important when it comes to new features.

The most important issues are Didas-git/Nekdis#6 which discusses the idea of a new entire relation system now that redis graph eol was announced, and Didas-git/Nekdis#10 which talks about a more front on/no hidden behaviour approach when it comes to indexing data.

[Didas-git]

V0.12 Changelog

This is one of the biggest changes in a while and im really proud of it, it brings lots of features, but i will also combine the 0.11 changelog here since i didn't write one previously.

• V0.12 Changelog
• Examples
• Array of objects!
• Schema
  • New type
  • New options
  • New Properties
  • Text
  • String
  • extends
  • literal
  • Shared Schemas
  • index defaults to false
• Document
• Model \& Search

Examples

I have worked on a example that takes advantage of a lot of the new features, it is a simple api that uses nekdis as its database and you can find its repository here.

The table example was also slightly updated.

Array of objects!

Array of objects are now fully supported in both JSON and HASH and are Search capable.

Schema

New type

In v0.12 i added the bigint type which allows you to well, use bigints. Just keep in mind that they are saved as a string in JSON and indexed as TAG to search.

New options

• Stop Words
  • You can pass an array of stop words to the stopWords option on the schema
• Word Stemming
  • You can change the stemming language by passing your chosen language to the language options.

New Properties

Text

text fields got 2 new options:

• phonetic - Choose the phonetic matcher for the field
• weight - Define the importance of the field

String

string fields now have an optional property called caseSensitive which if set to true will pass the CASESENSITIVE option when indexing the field

extends

Now schemas can extend another schema just like classes.

const baseSchema = client.schema({
    key: "string"
});

// userSchema has `key: string`
const userSchema = client.schema({
    name: "string",
    age: "number"
}).extends(baseSchema);

literal

literal is an option that can be passed into string, number and bigint types, they are the same as typescript's literal types.

const aSchema = client.schema({
    // Emulate an `enum` with number literals
    // `type` can only be `0`, `1`, `2` or `3`
    type: {type: "number", literal: [0, 1, 2, 3]}
})

Shared Schemas

object types now can accept schemas into their properties making it so that you can write 1 schema that will work for multiple objects.

This example was taken from one of the repositories i have shared above as example.

const sharedValueSecondsSchema = client.schema({
    value: "number",
    inSeconds: "number"
});

const weaponSchema = client.schema({
    hit: {
        type: "object",
        properties: {
            limit: {
                type: "number",
                optional: true
            },
            cooldown: {
                type: "object",
                properties: sharedValueSecondsSchema
            }
        }
    },
    duration: {
        type: "object",
        properties: sharedValueSecondsSchema
    },
});

index defaults to false

Now index will default to false, this is extremely important in my opinion because it saves time parsing the schema and doesn't use as much space in redis itself. Also when it comes to JSON some optimizations where made so that tuples do not need to be parsed if they aren't indexed.

Document

The documents received some changes, the biggest one being that all the internal parsing was remade to be both faster and to actually work with tuples, nested objects on hashes and array of objects.

There were other small but desired changes:

• No more clutter is saved to the document
  • The global_prefix, model_name, prefix properties are not saved anymore to the database since they were never actually needed
• The types are now expanded which makes them readable
  • If you come into some issues when it comes to not matching types please report since this might be to a currently open issue in typescript for mapped types
• Values now get saved even if not present on the schema
  • Keep in mind that they will also not be formatted so any errors are on you

Model & Search

Due to the new parsing logic to create indexes i was able to reduce memory usage by a substantial amount, this doesn't only provide well less memory usage but it also helps improve search's query building speed.

[net-tech]

This is exactly the kind of thing I'm looking for. It brings better TypScript types which I need to make sure that I don't miss a required property. It also allows for better intellisense and TypeScript errors when I fetch a document. 👍 from me.

[Didas-git]

V0.13 Changelog

• V0.13 Changelog
• Relations
  • Creating relations
  • First step
  • Second Step
  • Getting all the relations
  • How it works
• Redis-cli Example
• Nekdis example
• Installation
• Quality of life
• Fixes
• Know Bugs

Relations

As of Nekdis 0.13 a new option called enabledInjections was added, this allows the client to inject (add) lua scripts to your redis instance, this allows us to make relations Atomic & bring new features like atomic updates in the future.

As for the functionality of the relation script itself it is divided into 2 categories each one with its sub categories.

Creating relations

There are 2 commands to create relations JSONCR and HCR for JSON and HASH respectively, they receive exactly the same arguments in the same format:

FACLL JSONCR 3 inId outId omitId field meta

Where:

• inId - The id/key you are creating the relation for
• outId - The id/key you are relating to
• omitId - The id/key where the metadata will be stored
• field - The "field" where this relation is saved, its actually used to create the Set to save the omitted keys at.
• meta - The metadata is a stringified JSON object with the data you will use as metadata

First step

The first step of creating a relation is creating the key where all the metadata will be saved, the type of the key can be either JSON or HASH depending on the command you use.

Second Step

The second step is to append the id to the set that stores all the relations.

Internally all it does is SADD iniId:field omitId

Getting all the relations

The script also provides you a JSONGR and HGR function to fetch all of the relations of said field.

The syntax is:

FCALL JSONGR 1 key field

Where:

• key - The id/key that you want to fetch the relations of
• field - The field name of the relation

This is easier to see if you look at the cli examples.

How it works

The first step is to get all the omitted ids from the set using SMEMBERS so they can be processed.

Then we process them by fetching them, getting the out field and fetching the respective id/key.

Redis-cli Example

Lets setup 2 different users and create a relation between them with some metadata

JSON.SET user:1 $ '{"name": "DidaS"}'
JSON.SET user:2 $ '{"name": "Leibale"}'

FCALL JSONCR 3 user:1 user:2 user:contacts:1 contacts '{"company": "Redis"}'

After running this you will see 2 new keys on your database:

A JSON key called user:contacts:1 which will look like:

{
    "in": "user:1",
    "out": "user:2",
    "company": "Redis"
}

And a set called user:1:contacts which will contain user:contacts:1

To fetch all the relations you can run the following

FCALL JSONGR 1 user:1 contacts

Which for this example will return

[
    {
        "name": "Leibale"
    }
]

You can note that metadata is not appended to the returning object, this is because metadata only exists so you can leverage RediSearch functionality on relations.

An example of this would be:

FT.CREATE exampleIdx ON JSON PREFIX 1 user:contacts: SCHEMA $.in AS in TAG $.out AS out TAG $.company AS company TEXT

And then we want to search all contacts of user:1 that work on redis

FT.SEARCH exampleIdx '@in:{user\:1} @company:redis'

Nekdis example

This is a direct translation of the cli example into nekdis syntax with a bonus (explained on the comments)

const userSchema = client.schema({
    name: "string",
    contacts: {
        type: "relation",
        schema: "self",
        index: true,
        // Meta could be another schema just like the properties in an object type
        meta: {
            company: "text"
        }
    }
});

const userModel = client.model("User", userSchema);
// This will also create the relation indexes to be used in the constrains
await userModel.createIndex();

// Im not using "createAndSave" so i can have the document without having to create and then fetch
const user1 = userModel.create({
    name: "DidaS"
});

const user2 = userModel.create({
    name: "Leibale"
});

const user3 = userModel.create({
    name: "Webstrand"
});

await userModel.save(user1);
await userModel.save(user2);
await userModel.save(user3);

// Im passing the record id just to exemplify that you can use either the document or an id/key
await userModel.relate(user1).to(user2.$record_id).as("contacts").with({ company: "Redis" }).exec();
await userModel.relate(user1).to(user3).as("contacts").with({ company: "Unknown" }).exec();

await userModel.get(user1.$record_id, { withRelations: true });
/*
JSONDocument {
    name: "DidaS",
    contacts: [
        JSONDocument { name: "Leibale" },
        JSONDocument { name: "Webstrand" }
    ]
}
*/

await userModel.get(user1.$record_id, {
    withRelations: true,
    relationsConstrain: {
        contacts: (search) => search.where("in").eq(userModel.sanitize(user1.$record_id)).and("company").eq("redis")
    }
})
/*
JSONDocument {
    name: "DidaS",
    contacts: [
        JSONDocument { name: "Leibale" }
    ]
}
*/

// This is what the search on the example returns, the one above is what Nekdis does to make it better for the user
await userModel.get(user1.$record_id, {
    withRelations: true,
    returnMetadataOverRelation: true,
    relationsConstrain: {
        contacts: (search) => search.where("in").eq(userModel.sanitize(user1.$record_id)).and("company").eq("redis")
    }
})
/*
JSONDocument {
    name: "DidaS",
    contacts: [
        JSONDocument {
            company: "Redis",
            in: "Nekdis:V1:User:8cb985f8-2430-49a3-85a2-e57883d4fe45",
            out: "Nekdis:V1:User:49d110c0-14b0-4965-a27b-78b3480daa47"
        }
    ]
}
*/

Installation

This update is available as a beta version on npm, i will post a comment when i post it as the stable version

npm i nekdis@beta

Quality of life

There were some major quality of life improvements when it comes to the types, they should be more accurate and easier to understand now.

For the ones who might be interested in contributing, i improved some of the internal naming so now it should be easier to understand the type transformations

Fixes

• <Client>.connect will now wait for the client to connect before resolving
• object, tuple and array types now can be globally indexed

Know Bugs

In some scenarios you might get a typescript error saying the types are not compatible when they are, this is an issue regarding mapped types that will be fixed in TS 5.3

[Didas-git]

V0.13.1 Changelog

• Reverted a breaking change on the Model (suffix exists again on the ModelOptions)
• Library now exports more types but doesnt export the Schema class anymore
• Added <Model>.sanitize method so you can sanitize search strings (like record ids)
• in & out are now typed if you want to search on them
• Some internal name changes that dont matter much here

Final notes

0.13 will be made stable in 2/3 days after i merge it in this PR, i will also edit the previous changelog to correct the examples that are wrong (lack sanitization)

[Didas-git]

V0.13.2 -> V0.13.4 Changelog

• Fixed multiple indexing bugs
• Fixed cirtical bugs regarding document changes
• Fixed top level indexing
• Firther improved lua methods for relations
• Vectors have been temporarily disabled from HASH
• Changed export scheme