Fast Persistent Dictionary

Fast Persistent Dictionary, is an implementation of a persistent dictionary in C#. This dictionary-like data structure is designed to persist data to disk, serving as a reliable workhorse for cases where the application requires storing large amounts of data, but also needs to operate with high efficiency and minimize disk space usage.

The dictionary comes with a plethora of features and methods, giving the developer the flexibility to manipulate the data as per their requirements. Some of the key features include options to add or update a record, clear all records, load or save the dictionary, or perform common mathematical operations across the dataset. The dictionary also supports compression to help manage disk usage and provides a GROBUF Serializer for data serialization, facilitating efficient storage of complex objects.

This dictionary makes use of the .NET IEnumerable and ISerializable interfaces, allowing it to be used in various scenarios where these interfaces are required, for example, in .NET collections or serialization operations.

High frequency updates are managed via a timer which triggers a background thread at a regular interval defined by the user. This enhances the dictionary's performance by offloading tasks that do not need to happen immediately, such as writing to disk.

The project also provides exception handling and recovery features. A dispose mechanism is incorporated ensuring that all resources are released when the instance is no longer required, enhancing the project’s stability and resilience.

Fast Persistent Dictionary is an open-source project crafted with care, providing a robust and efficient persistent dictionary in C#. It's an invaluable tool when building applications requiring efficient, reliable, and persistent data storage.

Fast Persistent Dictionary Features

• Single File Database: the In use database and the saved and loadable format is all compiled in a single file.
• Performance: Fast Persistent Dictionary supports a high rate of updates and retrieves. Typically surpassing ESENT by a wide margin.
• Simplicity: a FastPersistentDictionary looks and behaves like the .NET Dictionary class. No extra method calls are required.
• Concurrency: each data structure can be accessed by multiple threads.
• Scale: Values can be up to 2gb in size.
• No Serialization Flags: Any key or value can be used as long as it is serializable by Grobuf.

Benchmarks

BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4037/23H2/2023Update/SunValley3)
Intel Core i7-10875H CPU 2.30GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK 8.0.200
  [Host]     : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2 [AttachedDebugger]
  Job-URLKUQ : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2

InvocationCount=1  UnrollFactor=1  

Quick Start Demo

Setting up a FastPersistentDictionary is simple and straightforward and matches the implementation of a standard Dictionary in C# as much as possible with various additional extension methods.

int N = 100000;
private FastPersistentDictionary<string, string> fastPersistentDictionary= new FastPersistentDictionary<string, string>(path: fastPersistentDictionary);
//Strings for example but can work with any type.
 for (int i = 0; i < N; i++)
 {
     string key = $"Key{i}";
     string value = $"Value{i}";
     FastPersistentDictionary.Add(key, value); 
 }

 for (int i = 0; i < N; i++)
 {
     string key = $"Key{i}";
     string value;
     bool found = FastPersistentDictionary.TryGetValue(key, out value); 
 }

Table of Contents

• Project Title
• Quick Start Demo
• Table of Contents
• Usage
• Contribute
• License

Usage

(Back to top)

FastPersistentDictionary

The FastPersistentDictionary is a highly efficient and flexible persistent dictionary for C# that provides a range of dictionary functionalities while ensuring data persistence on disk. Developed by James Grice, this library is designed to be utilized just like any other dictionary, but with the added benefit of persistency and reduced disk space usage.

Installation

To install the FastPersistentDictionary library, you can download it from GitHub and include it in your project.

Quick Start Guide

1. Creating a FastPersistentDictionary

Here's a simple example to create a FastPersistentDictionary:

using FastPersistentDictionary;

var dictionary = new FastPersistentDictionary<string, int>("path/to/your/dictionary/file");

2. Adding Entries

To add entries to the dictionary:

dictionary.Add("key1", 100);
dictionary.Add("key2", 200);

3. Accessing Entries

The FastPersistentDictionary can be accessed like a regular dictionary:

int value = dictionary["key1"];
Console.WriteLine(value);  // Output: 100

4. Updating Entries

Updating existing entries is straightforward:

dictionary["key1"] = 150;

5. Checking Existence of Keys and Values

You can check if a key or a value exists in the dictionary:

bool hasKey = dictionary.ContainsKey("key1");    // true
bool hasValue = dictionary.ContainsValue(200);   // true

6. Removing Entries

Removing entries from the dictionary:

dictionary.Remove("key1");

7. Iterating Over Entries

You can easily iterate over the entries in the dictionary:

foreach (var kvp in dictionary)
{
    Console.WriteLine($"{kvp.Key} : {kvp.Value}");
}

8. Compacting the Database

To minimize the file size on the disk manually, you can compact the database:

dictionary.CompactDatabaseFile();

9. Saving and Loading the Dictionary

You can save the current state of the dictionary to a file:

dictionary.SaveDictionary("path/to/save/file");

And you can load an existing dictionary from a file:

var loadedDictionary = new FastPersistentDictionary<string, int>("path/to/save/file");
loadedDictionary.LoadDictionary("path/to/save/file");

10. Disposing the Dictionary

Make sure to dispose of the dictionary properly when it is no longer needed:

dictionary.Dispose();

Advanced Features

1. Bulk Operations

You can perform bulk-read operations:

var keys = new[] { "key1", "key2" };
var values = dictionary.GetBulk(keys);

2. Mathematical Operations

The FastPersistentDictionary supports mathematical operations:

int min = dictionary.Min();
int sum = dictionary.Sum();
int max = dictionary.Max();
double average = dictionary.Average(kvp => kvp.Value);

3. Query and Filter

Advanced querying, filtering, and extracting subsets of the dictionary:

var subset = dictionary.GetSubset((key, value) => value > 100);

bool any = dictionary.Any(kvp => kvp.Value > 150);
bool all = dictionary.All(kvp => kvp.Value > 50);

4. Importing Other Dictionaries

Import entries from another dictionary:

var anotherDictionary = new Dictionary<string, int>
{
    { "key3", 300 },
    { "key4", 400 }
};
dictionary.Merge(anotherDictionary);

Summary

The FastPersistentDictionary offers a wide range of functionalities for persistent data storage with minimal overhead on disk space. With the ease of use of any other dictionary and options for advanced operations, it is a powerful tool for large datasets and projects in C#.

Contribute

(Back to top)

Your support is invaluable and truly welcomed! Here's how you can contribute:

• Write Tests and Benchmarks: Enhance code reliability and performance evaluation.
• Enhance Documentation: Assist others in comprehending and effectively using FastPersistentDictionary.
• Submit Feature Requests and Bug Reports: Suggest new ideas and report issues to help refine FastPersistentDictionary.
• Optimize Performance: Offer optimizations and improvements to existing features..

License

(Back to top)

MIT license