Add IDBIndex [IndexedDB]

[RonPeters]

There does not seem to be a way to access the indexes in IndexedDB. In JavaScript, you would do something like this:

const myIndex = objectStore.index("myIndexedField"); // index name
const getRequest = myIndex.get(2); // index value to match

Would it be possible to add this interface? Probably something more like objectStore.GetIndex(indexName)

https://developer.mozilla.org/en-US/docs/Web/API/IDBIndex

Also IDBKeyRange and IDBCursor would be awesome and would solve my use cases.

[LostBeard]

@RonPeters I am always happy to add and fix interfaces to make BlazorJS more complete. I will see what I can do.

[RonPeters]

@LostBeard As always, I love your responsiveness. One more thing, there was no API binding to create the index on the object store, so I was doing this:

messageStore.JSRef!.Call<IDBRequest>("createIndex", indexName, keyPath, new { unique = true/false });

Not sure if that was the correct way to do it, but it should ideally have a binding as well.

[LostBeard]

I was missing quite a bit of the IndexedDB API. I have been working on filling it in. Almost done.

messageStore.JSRef!.Call<IDBRequest>("createIndex", indexName, keyPath, new { unique = true/false });

Yup. Using the JSObject.JSRef property is the way to go if the JSObject is missing a property or method.

[LostBeard]

Version 2.2.78 is up with updates to the IndexedDB API. I am not done going over it but I made a lot of progress that is ready to use.

Here is some test code that is working. I used it to work out some of the features. Some of the interfaces are a little vague and required testing and I'm sure I am not accounting for all use cases. I like to keep things as strongly typed as possible.

Most of the methods that return an IDBRequest have Async variants that return a Task where TResult is the result from the IDBRequest when it completes, or they throw if it errors.

Let me know if you find any issues, missing parts, etc. with the API.

var dbName = "garden";
var dbStoreName = "fruit";
var idb = await IDBDatabase.OpenAsync(dbName, 2, (evt) =>
{
    // upgrade needed
    try
    {
        JS.Set("_IDBVersionChangeEvent", evt);
        var oldVersion = evt.OldVersion;
        var newVersion = evt.NewVersion;
        using var request = evt.Target;
        using var db = request.Result;
        var stores = db.ObjectStoreNames;
        if (!stores.Contains(dbStoreName))
        {
            using var store = db.CreateObjectStore<string, Fruit>(dbStoreName, new IDBObjectStoreCreateOptions { KeyPath = "color" });
            store.CreateIndex<string>("name_db", "name");
        }
    }
    catch (Exception ex)
    {
        var nmt = true;
    }
});
// transaction
using var tx = idb.Transaction(dbStoreName, "readwrite");
using var objectStore = tx.ObjectStore<string, Fruit>(dbStoreName);

// add some data
await objectStore.PutAsync(new Fruit { Name = "apple", Color = "red" });
await objectStore.PutAsync(new Fruit { Name = "orange", Color = "orange" });

// count
var count = await objectStore.CountAsync();
JS.Log("count", count);

// index names
var indexNames = objectStore.IndexNames;
JS.Log("indexNames", indexNames);

// get an IDBIndex
var myIndex = objectStore.Index("name_db");

// get on IDBIndex. returns null if not found.
var get = await myIndex.GetAsync("orange");
JS.Log("get", get);

var getNotFound = await myIndex.GetAsync("cucumber");
JS.Log("getNotFound", getNotFound);

// getKey on IDBIndex. returns null if not found.
var getKey = await myIndex.GetKeyAsync("orange");
JS.Log("getKey", getKey);

// getAll on IDBIndex
var getAll = await myIndex.GetAllAsync();
JS.Log("getAll", getAll);

// getAll on ObjectStore
var getAllStore = await objectStore.GetAllAsync();
JS.Log("getAllStore", getAllStore);

// IDBCursor iteration
using var cursor = await myIndex.OpenCursorAsync();
var hasData = cursor != null;
while (hasData)
{
    JS.Log("Entry", cursor!.Value);
    hasData = await cursor!.ContinueAsync();
}
JS.Log("Done");

// getAllKeys
var keys = await myIndex.GetAllKeysAsync();
JS.Log("keys", keys);

// delete all
foreach(var key in keys)
{
    await objectStore.DeleteAsync(key);
}

// should be empty now
var keysEmpty = await myIndex.GetAllKeysAsync();
JS.Log("keysEmpty", keysEmpty);

[RonPeters]

btw, I think the new IDBTransactionMode.readonly has a conflict because it's a reserved keyword. You end up having to use IDBTransactionMode.@readonly. Maybe camel case it?

[LostBeard]

If you look at the signature I created for IDBDatabase.Transaction you can see it uses EnumString. EnumString is a type I created to allow serializing/deserializing Enums to/from their string values instead of their integer values.

IDBTransactionMode.@readonly serializes to "readonly". If I camel case it to "IDBTransactionMode.ReadOnly" , it would serialize to "ReadOnly". I could make it use lowercase when serialized, but then it would not work for strings that have upper case characters... unless I added attribute flags to allow indicating how to handle character case when serializing.

IDBDatabase.Transaction signature

IDBTransaction Transaction(Union<string, IEnumerable<string>> storeNames, EnumString<IDBTransactionMode> mode)

EnumString has implicit operators that can convert between string <-> EnumString<Enum> and the Enum <-> EnumString<Enum>.

Therefore, both of the below lines are equal and both will work.

using var tx = idb.Transaction(dbStoreName, IDBTransactionMode.@readonly);
using var tx = idb.Transaction(dbStoreName, "readonly");

// "readonly" is the default mode (Javascript spec), so this is also the same as the two above
using var tx = idb.Transaction(dbStoreName);

I see it is working as expected. Is your comment due to personal preference or did you find an issue?

[LostBeard]

EnumString usage in BlazorJS will be removed in the next update. Please just use the string values.

The update will also have improved IDBTransaction interface.

[RonPeters]

It is working as expected. I thought perhaps it was a bug in the mapping, but now I understand your design choice. What about just plain string consts wrapped in a static class instead? I look forward to your next update. This library is a lifesaver!

[LostBeard]

What about just plain string consts wrapped in a static class instead?

I have/am considering that.

[LostBeard]

Version 2.2.79 is up. I did another quick look through the IndexedDB API and finished what was left, including (most of) the in-code documentation. 👍 I'll have more time during the week to give it a more thorough look.

I am going to close this issue as completed. Continue posting here or in a new issue if needed.

[RonPeters]

Trying to use compound/composite indexes and running into an issue:

In OnUpgradeNeeded objectStore.CreateIndex<(byte[], long)>(indexName, "id, lastModified");

then:

using var index = objectStore.Index(indexName);
using var range = IDBKeyRange<(byte[], long)>.Bound((idBytes, 0), (idBytes, long.MaxValue));
using var cursor = await index.OpenCursorAsync(range);  <-- cannot convert from 'SpawnDev.BlazorJS.JSObjects.IDBKeyRange<(byte[], long)>' to 'SpawnDev.BlazorJS.Union<SpawnDev.BlazorJS.JSObjects.IDBKeyRange<byte[]>, byte[]>?'

I can't apply the type (<(byte[], long)>) to the OpenCursor or the Index(indexName).

Example: https://stackoverflow.com/questions/12084177/in-indexeddb-is-there-a-way-to-make-a-sorted-compound-query

[RonPeters]

The problem is IDBObjectStore.Index assumes the index uses the same TKey as the IDBObjectStore. You should be able to specify a different type for the index key, like TIndexKey in CreateIndex

I think: public IDBIndex<TIndexKey, TValue> Index<TIndexKey>(string name) => JSRef.Call<IDBIndex<TIndexKey, TValue>>("index", name);

[LostBeard]

Fixed. Version 2.2.80 is up. Thanks for the help.

[LostBeard]

I see you are using a type of (byte[], long), a ValueTuple. I just did a test to see if Blazor can serialize that and it does not (please correct me if I am wrong.) In my test it serializes to an empty object. {}.

How are you expecting that tuple to serialize?

byte[] serializes to Uint8Array. So maybe it would serialize to an array like [ Uint8Array, Number ]? A new JsonConverter should be able to handle serialization and deserialization of Tuples/ValueTuples.

[RonPeters]

That is a fantastic question. I didn't yet have my actual data ready, so I did not fully test it. After thinking about it, I'm not sure if there is a type safe way to do it. I think a JsonConverter would 1. add field names and 2. serialize the bytes to string. I think I'm just going to try object[] and bail on type safety for the moment.

[LostBeard]

I think a JsonConverter would 1. add field names and 2. serialize the bytes to string.

The IJSRuntime, which is what BlazorJSRuntime from BlazorJS uses, serializes byte[] to Uint8Array due to the JsonSerializer options used for Javascript interop. If the field names were added, they would be "Item1", "Item2", etc. I think serializing to an array of the Tuple's values is more flexible and efficient.

I just uploaded version 2.2.81. I added serialization and deserialization support for ValueTuple (Ex. (byte[], long)), and Tuple (Ex. Tuple<byte[], long>). Both Tuple and ValueTuple serialize to an array of the tuple's values.

Serialization ex: In C# set a Javascript global variable to the tuple calue

(byte[], long)? valueTuple = (new byte[] { 1, 2, 3 }, (long)5);
JS.Set("valueTuple", valueTuple);

In Javascript read the tuple as an array

// below prints "valueTuple[0] Uint8Array"
console.log('valueTuple[0]', valueTuple[0].constructor.name);
// below prints "valueTuple[1] Number"
console.log('valueTuple[1]', valueTuple[1].constructor.name);

ValueTuple IDBKeyRange Test

And I ran some tests based on your posted code. I used the test code below to check if an index of (byte[], long) would work as expected and, with the new Tuple JsonConverters I added, it does. 👍 Hopefully this helps you out when you are ready to use it.

Test class that will be stored.

public class Fruit
{
    public (byte[], long) MyKey { get; set; }
    public string Name { get; set; }
    public string Color { get; set; }
}

var dbName = "garden_tuple";
var dbStoreName = "fruit";
var idb = await IDBDatabase.OpenAsync(dbName, 2, (evt) =>
{
    // upgrade needed
    using var request = evt.Target;
    using var db = request.Result;
    var stores = db.ObjectStoreNames;
    if (!stores.Contains(dbStoreName))
    {
        using var store = db.CreateObjectStore<string, Fruit>(dbStoreName, new IDBObjectStoreCreateOptions { KeyPath = "name" });
        store.CreateIndex<(byte[], long)>("tuple_index", "myKey");
    }
});

// transaction
using var tx = idb.Transaction(dbStoreName, "readwrite");
using var objectStore = tx.ObjectStore<string, Fruit>(dbStoreName);

// add some data
await objectStore.PutAsync(new Fruit { Name = "apple", Color = "red", MyKey = (new byte[] { 1, 2, 3 }, 5) });
await objectStore.PutAsync(new Fruit { Name = "orange", Color = "orange", MyKey = (new byte[] { 1, 2, 5 }, 5) });
await objectStore.PutAsync(new Fruit { Name = "lemon", Color = "yellow", MyKey = (new byte[] { 1, 2, 5 }, 5) });
await objectStore.PutAsync(new Fruit { Name = "lime", Color = "green", MyKey = (new byte[] { 33, 33, 45 }, 5) });

// get an IDBIndex
using var myIndex = objectStore.Index<(byte[], long)>("tuple_index");

// create a range using ValueTuple type
using var range = IDBKeyRange<(byte[], long)>.Bound((new byte[] { 0, 0, 0 }, 0), (new byte[] { 5, 5, 5 }, long.MaxValue));

// getAll on IDBIndex using the above range
using var getAll = await myIndex.GetAllAsync(range);

// below prints "apple", "orange", "lemon"
// the "lime" entry's byte[] is outside of our range and therefore not included
foreach (var item in getAll.ToArray())
{
    JS.Log(item.Name);
}

[LostBeard]

You can test how comparisons will work using IDBFactory.Cmp instance method.

// new IDBFactory() is the global indexedDB instance
using var idbFactory = new IDBFactory(); 
var cmpRet0 = idbFactory.Cmp<(byte[], long)>((new byte[] { 1, 2, 3 }, 6), (new byte[] { 1, 2, 3 }, 5));
// cmpRet0 == 1
var cmpRet1 = idbFactory.Cmp<(byte[], long)>((new byte[] { 1, 2, 3 }, 5), (new byte[] { 1, 2, 3 }, 5));
// cmpRet1 == 0
var cmpRet2 = idbFactory.Cmp<(byte[], long)>((new byte[] { 1, 2, 2 }, 5), (new byte[] { 1, 2, 3 }, 5));
// cmpRet2 == -1
var cmpRet3 = idbFactory.Cmp<(byte[], long)>((new byte[] { 1, 2, 4 }, 5), (new byte[] { 1, 2, 3 }, 4));
// cmpRet3 == 1

[LostBeard]

Or test the range.

using var range = IDBKeyRange<(byte[], long)>.Bound((new byte[] { 0, 0, 0 }, 0), (new byte[] { 5, 5, 5 }, long.MaxValue));
var included = range.Includes((new byte[] { 1, 2, 4 }, 5));
// included  == true