search

redis / redis-om-dotnet

Object mapping, and more, for Redis and .NET
MIT License
469 stars 79 forks source link
dotnet linq redis

readme



Redis OM

Object mapping, and more, for Redis and .NET

NuGet License Build Status

Redis OM .NET makes it easy to model Redis data in your .NET Applications.

Redis OM .NET | Redis OM Node.js | Redis OM Spring | Redis OM Python

Table of contents - [💡 Why Redis OM?](#-why-redis-om) - [💻 Installation](#-installation) - [🏁 Getting started](#-getting-started) - [Starting Redis](#starting-redis) - [📇 Modeling your domain (and indexing it!)](#-modeling-your-domain-and-indexing-it) - [🔑 Keys and Ids](#-keys-and-ids) - [🔎 Querying](#-querying) - [🖩 Aggregations](#-aggregations) - [📚 Documentation](#-documentation) - [⛏️ Troubleshooting](#-troubleshooting) - [✨ Redis Stack](#-redis-stack) - [Why this is important](#why-this-is-important) - [So how do you get Redis Stack?](#so-how-do-you-get-redis-stack) - [❤️ Contributing](#-contributing) - [❤️ Our Contributors](#-our-contributors)

💡 Why Redis OM?

Redis OM provides high-level abstractions for using Redis in .NET, making it easy to model and query your Redis domain objects.

This preview release contains the following features:

💻 Installation

Using the dotnet cli, run:

dotnet add package Redis.OM

🏁 Getting started

Starting Redis

Before writing any code you'll need a Redis instance with the appropriate Redis modules! The quickest way to get this is with Docker:

docker run -p 6379:6379 -p 8001:8001 redis/redis-stack

This launches the redis-stack an extension of Redis that adds all manner of modern data structures to Redis. You'll also notice that if you open up http://localhost:8001 you'll have access to the redis-insight GUI, a GUI you can use to visualize and work with your data in Redis.

📇 Modeling your domain (and indexing it!)

With Redis OM, you can model your data and declare indexes with minimal code. For example, here's how we might model a customer object:

[Document(StorageType = StorageType.Json)]
public class Customer
{
   [Indexed] public string FirstName { get; set; }
   [Indexed] public string LastName { get; set; }
   public string Email { get; set; }
   [Indexed(Sortable = true)] public int Age { get; set; }
   [Indexed] public string[] NickNames {get; set;}
}

Notice that we've applied the Document attribute to this class. We've also specified that certain fields should be Indexed.

Now we need to create the Redis index. So we'll connect to Redis and then call CreateIndex on an IRedisConnection:

var provider = new RedisConnectionProvider("redis://localhost:6379");
provider.Connection.CreateIndex(typeof(Customer));

Redis OM provides limited support for schema migration at this time. You can check if the index definition in Redis matches your current index definition using the IsIndexCurrent method on the RedisConnection. Then you may use that output to determine when to re-create your indexes when your types change. An example implementation of this would look like:

var provider = new RedisConnectionProvider("redis://localhost:6379");
var definition = provider.Connection.GetIndexInfo(typeof(Customer));

if (!provider.Connection.IsIndexCurrent(typeof(Customer)))
{
    provider.Connection.DropIndex(typeof(Customer));
    provider.Connection.CreateIndex(typeof(Customer));
}

Indexing Embedded Documents

There are two methods for indexing embedded documents with Redis.OM, an embedded document is a complex object, e.g. if our Customer model had an Address property with the following model:

[Document(IndexName = "address-idx", StorageType = StorageType.Json)]
public partial class Address
{
    public string StreetName { get; set; }
    public string ZipCode { get; set; }
    [Indexed] public string City { get; set; }
    [Indexed] public string State { get; set; }
    [Indexed(CascadeDepth = 1)] public Address ForwardingAddress { get; set; }
    [Indexed] public GeoLoc Location { get; set; }
    [Indexed] public int HouseNumber { get; set; }
}

Index By JSON Path

You can index fields by JSON path, in the top level model, in this case Customer you can decorate the Address property with an Indexed and/or Searchable attribute, specifying the JSON path to the desired field:

[Document(StorageType = StorageType.Json)]
public class Customer
{
   [Indexed] public string FirstName { get; set; }
   [Indexed] public string LastName { get; set; }
   public string Email { get; set; }
   [Indexed(Sortable = true)] public int Age { get; set; }
   [Indexed] public string[] NickNames {get; set;}
   [Indexed(JsonPath = "$.ZipCode")]
   [Searchable(JsonPath = "$.StreetAddress")]
   public Address Address {get; set;}
}
Indexing Arrays of Objects

This methodology can also be used for indexing string and string-like value-types within objects within Arrays and Lists, so for example if we had an array of Addresses, and we wanted to index the cities within those addresses we could do so with the following

[Indexed(JsonPath = "$.City")]
public Address[] Addresses { get; set; }

Those Cities can then be queried with an Any predicate within the main Where clause.

collection.Where(c=>c.Addresses.Any(a=>a.City == "Satellite Beach"))
Limitations

The way Redis indexes fields within a collection of embedded objects does not allow multiple predictates to be specified to a given document e.g.

collection.Where(c=>c.Addresses.Any(a=>a.City == "Satellite Beach" && a.ZipCode == "32937))

In the above case the query can only check if the Addresses collection contains an entry that is Satellite Beach, and Contains an entry that has a zip code of 32937, rather than an entry that has both the city of Satellite Beach and a zip code of `32937

Cascading Index

Alternatively, you can also embedded models by cascading indexes. In this instance you'd simply decorate the property with Indexed and set the CascadeDepth to whatever to however may levels you want the model to cascade for. The default is 0, so if CascadeDepth is not set, indexing an object will be a no-op:

[Document(StorageType = StorageType.Json)]
public class Customer
{
   [Indexed] public string FirstName { get; set; }
   [Indexed] public string LastName { get; set; }
   public string Email { get; set; }
   [Indexed(Sortable = true)] public int Age { get; set; }
   [Indexed] public string[] NickNames {get; set;}
   [Indexed(CascadeDepth = 2)]
   public Address Address {get; set;}
}

In the above case, all indexed/searchable fields in Address will be indexed down 2 levels, so the ForwardingAddress field in Address will also be indexed.

Once the index is created, we can:

Let's see how!

Indexing DateTimes

As of version 0.4.0, all DateTime objects are indexed as numerics, and they are inserted as numerics into JSON documents. Because of this, you can query them as if they were numerics!

🔑 Keys and Ids

ULIDs and strings

Ids are unique per object, and are used as part of key generation for the primary index in Redis. The natively supported Id type in Redis OM is the ULID. You can bind ids to your model, by explicitly decorating your Id field with the RedisIdField attribute:

[Document(StorageType = StorageType.Json)]
public class Customer
{
    [RedisIdField] public Ulid Id { get; set; }
    [Indexed] public string FirstName { get; set; }
    [Indexed] public string LastName { get; set; }
    public string Email { get; set; }
    [Indexed(Sortable = true)] public int Age { get; set; }
    [Indexed] public string[] NickNames { get; set; }
}

When you call Set on the RedisConnection or call Insert in the RedisCollection, to insert your object into Redis, Redis OM will automatically set the id for you and you will be able to access it in the object. If the Id type is a string, and there is no explicitly overriding IdGenerationStrategy on the object, the ULID for the object will bind to the string.

Other types of ids

Redis OM also supports other types of ids, ids must either be strings or value types (e.g. ints, longs, GUIDs etc. . .), if you want a non-ULID id type, you must either set the id on each object prior to insertion, or you must register an IIdGenerationStrategy with the DocumentAttribute class.

Register IIdGenerationStrategy

To Register an IIdGenerationStrategy with the DocumentAttribute class, simply call DocumentAttribute.RegisterIdGenerationStrategy passing in the strategy name, and the implementation of IIdGenerationStrategy you want to use. Let's say for example you had the StaticIncrementStrategy, which maintains a static counter in memory, and increments ids based off that counter:

public class StaticIncrementStrategy : IIdGenerationStrategy
{
    public static int Current = 0;
    public string GenerateId()
    {
        return (Current++).ToString();
    }
}

You would then register that strategy with Redis.OM like so:

DocumentAttribute.RegisterIdGenerationStrategy(nameof(StaticIncrementStrategy), new StaticIncrementStrategy());

Then, when you want to use that strategy for generating the Ids of a document, you can simply set the IdGenerationStrategy of your document attribute to the name of the strategy.

[Document(IdGenerationStrategyName = nameof(StaticIncrementStrategy))]
public class ObjectWithCustomIdGenerationStrategy
{
    [RedisIdField] public string Id { get; set; }
}

Key Names

The key names are, by default, the fully qualified class name of the object, followed by a colon, followed by the Id. For example, there is a Person class in the Unit Test project, an example id of that person class would be Redis.OM.Unit.Tests.RediSearchTests.Person:01FTHAF0D1EKSN0XG67HYG36GZ, because Redis.OM.Unit.Tests.RediSearchTests.Person is the fully qualified class name, and 01FTHAF0D1EKSN0XG67HYG36GZ is the ULID (the default id type). If you want to change the prefix (the fully qualified class name), you can change that in the DocumentAttribute by setting the Prefixes property, which is an array of strings e.g.

[Document(Prefixes = new []{"Person"})]
public class Person

Note: At this time, Redis.OM will only use the first prefix in the prefix list as the prefix when creating a key name. However, when an index is created, it will be created on all prefixes enumerated in the Prefixes property

🔎 Querying

We can query our domain using expressions in LINQ, like so:

var customers = provider.RedisCollection<Customer>();

// Insert customer
customers.Insert(new Customer()
{
    FirstName = "James",
    LastName = "Bond",
    Age = 68,
    Email = "bondjamesbond@email.com"
});

// Find all customers whose last name is "Bond"
customers.Where(x => x.LastName == "Bond");

// Find all customers whose last name is Bond OR whose age is greater than 65
customers.Where(x => x.LastName == "Bond" || x.Age > 65);

// Find all customers whose last name is Bond AND whose first name is James
customers.Where(x => x.LastName == "Bond" && x.FirstName == "James");

// Find all customers with the nickname of Jim
customers.Where(x=>x.NickNames.Contains("Jim"));

Vectors

Redis OM .NET also supports storing and querying Vectors stored in Redis.

A Vector<T> is a representation of an object that can be transformed into a vector by a Vectorizer.

A VectorizerAttribute is the abstract class you use to decorate your Vector fields, it is responsible for defining the logic to convert the object's that Vector<T> is a container for into actual vector embeddings needed. In the package Redis.OM.Vectorizers we provide vectorizers for HuggingFace, OpenAI, and AzureOpenAI to allow you to easily integrate them into your workflows.

Define a Vector in your Model.

To define a vector in your model, simply decorate a Vector<T> field with an Indexed attribute which defines the algorithm and algorithmic parameters and a Vectorizer attribute which defines the shape of the vectors, (in this case we'll use OpenAI):

[Document(StorageType = StorageType.Json)]
public class OpenAICompletionResponse
{
    [RedisIdField]
    public string Id { get; set; }

    [Indexed(DistanceMetric = DistanceMetric.COSINE, Algorithm = VectorAlgorithm.HNSW, M = 16)]
    [OpenAIVectorizer]
    public Vector<string> Prompt { get; set; }

    public string Response { get; set; }

    [Indexed]
    public string Language { get; set; }

    [Indexed]
    public DateTime TimeStamp { get; set; }
}

Insert Vectors into Redis

With the vector defined in our model, all we need to do is create Vectors of the generic type, and insert them with our model. Using our RedisCollection, you can do this by simply using Insert:

var collection = _provider.RedisCollection<OpenAICompletionResponse>();
var completionResult = new OpenAICompletionResponse
{
    Language = "en_us", 
    Prompt = Vector.Of("What is the Capital of France?"), 
    Response = "Paris", 
    TimeStamp = DateTime.Now - TimeSpan.FromHours(3)
};
collection.Insert(completionResult);

The Vectorizer will manage the embedding generation for you without you having to intervene.

Query Vectors in Redis

To query vector fields in Redis, all you need to do is use the VectorRange method on a vector within our normal LINQ queries, and/or use the NearestNeighbors with whatever other filters you want to use, here's some examples:

var prompt = "What really is the Capital of France?";

// simple vector range, find first within .15
var result = collection.First(x => x.Prompt.VectorRange(prompt, .15));

// simple nearest neighbors query, finds first nearest neighbor
result = collection.NearestNeighbors(x => x.Prompt, 1, prompt).First();

// hybrid query, pre-filters result set for english responses, then runs a nearest neighbors search.
result = collection.Where(x=>x.Language == "en_us").NearestNeighbors(x => x.Prompt, 1, prompt).First();

// hybrid query, pre-filters responses newer than 4 hours, and finds first result within .15
var ts = DateTimeOffset.Now - TimeSpan.FromHours(4);
result = collection.First(x=>x.TimeStamp > ts && x.Prompt.VectorRange(prompt, .15));

What Happens to the Embeddings?

With Redis OM, the embeddings can be completely transparent to you, they are generated and bound to the Vector<T> when you query/insert your vectors. If however you needed your embedding after the insertion/Query, they are available at Vector<T>.Embedding, and be queried either as the raw bytes, as an array of doubles or as an array of floats (depending on your vectorizer).

Configuration

The Vectorizers provided by the Redis.OM.Vectorizers package have some configuration parameters that it will pull in either from your appsettings.json file, or your environment variables (with your appsettings taking precedence).

Configuration Parameter Description
REDIS_OM_HF_TOKEN HuggingFace Authorization token.
REDIS_OM_OAI_TOKEN OpenAI Authorization token
REDIS_OM_OAI_API_URL OpenAI URL
REDIS_OM_AZURE_OAI_TOKEN Azure OpenAI api key
REDIS_OM_AZURE_OAI_RESOURCE_NAME Azure resource name
REDIS_OM_AZURE_OAI_DEPLOYMENT_NAME Azure deployment

Semantic Caching

Redis OM also provides the ability to use Semantic Caching, as well as providers for OpenAI, HuggingFace, and Azure OpenAI to perform semantic caching. To use a Semantic Cache, simply pull one out of the RedisConnectionProvider and use Store to insert items, and GetSimilar to retrieve items. For example:

var cache = _provider.OpenAISemanticCache(token, threshold: .15);
cache.Store("What is the capital of France?", "Paris");
var res = cache.GetSimilar("What really is the capital of France?").First();

ML.NET Based Vectorizers

We also provide the packages Redis.OM.Vectorizers.ResNet18 and Redis.OM.Vectorizers.AllMiniLML6V2 which have embedded models / ML Pipelines in them to allow you to easily Vectorize Images and Sentences respectively without the need to depend on an external API.

🖩 Aggregations

We can also run aggregations on the customer object, again using expressions in LINQ:

// Get our average customer age
customerAggregations.Average(x => x.RecordShell.Age);

// Format customer full names
customerAggregations.Apply(x => string.Format("{0} {1}", x.RecordShell.FirstName, x.RecordShell.LastName),
      "FullName");

// Get each customer's distance from the Mall of America
customerAggregations.Apply(x => ApplyFunctions.GeoDistance(x.RecordShell.Home, -93.241786, 44.853816),
      "DistanceToMall");

📚 Documentation

This README just scratches the surface. You can find a full tutorial on the redis.io. All the summary docs for this library can be found on the repo's github page.

⛏️ Troubleshooting

If you run into trouble or have any questions, we're here to help!

First, check the FAQ. If you don't find the answer there, hit us up on the Redis Discord Server.

✨ Redis Stack

Redis OM can be used with regular Redis for Object mapping and getting objects by their IDs. For more advanced features like indexing, querying, and aggregation, Redis OM is dependent on the Redis Stack platform, a collection of modules that extend Redis.

Why this is important

Without Redis Stack, you can still use Redis OM to create declarative models backed by Redis.

We'll store your model data in Redis as Hashes, and you can retrieve models using their primary keys.

So, what won't work without Redis Stack?

  1. You won't be able to nest models inside each other.
  2. You won't be able to use our expressive queries to find object -- you'll only be able to query by primary key.

So how do you get Redis Stack?

You can use Redis Stack with your self-hosted Redis deployment. Just follow the instructions for Installing Redis Stack.

Don't want to run Redis yourself? Redis Stack is also available on Redis Cloud. Get started here.

❤️ Contributing

We'd love your contributions! If you want to contribute please read our Contributing document.

❤️ Our Contributors