throws System.NullReferenceException when putting many items with key comparer

[fangliuwh]

Hi, could you please have a look at this issue:

I encountered the following exception when I try to put many items with a key comparer. System.NullReferenceException was unhandled by user code HResult=-2147467261 Message=Object reference not set to an instance of an object. Source=LightningDB StackTrace: at LightningDB.Native.LmdbMethods.mdb_put(IntPtr txn, UInt32 dbi, ValueStructure& key, ValueStructure& data, PutOptions flags) at LightningDB.Native.Lmdb.mdb_put(IntPtr txn, UInt32 dbi, Byte[] key, Byte[] value, PutOptions flags) at TestLocalStore.cs:line 46 InnerException:

The code looks like this:

        Func<int, int, int> comparison = (l, r) => l.CompareTo(r);
        var options = new DatabaseConfiguration { Flags = DatabaseOpenFlags.Create };
        Func<byte[], byte[], int> compareWith = (l, r) => comparison(BitConverter.ToInt32(l, 0), BitConverter.ToInt32(r, 0));
        options.CompareWith(Comparer<byte[]>.Create(new Comparison<byte[]>(compareWith)));

        var _env = new LightningEnvironment(envName);
        _env.Open();

        using (var txnT = _env.BeginTransaction())
        using(var db1 = txnT.OpenDatabase(configuration: options))
        {
            txnT.DropDatabase(db1);
            txnT.Commit();
        }

        var txn = _env.BeginTransaction();
        var db = txn.OpenDatabase(configuration: options);

        var keysUnsorted = Enumerable.Range(1, 10000).Randomize().ToList();
        var keysSorted = keysUnsorted.ToArray();
        Array.Sort(keysSorted, new Comparison<int>(comparison));

        for (var i = 0; i < keysUnsorted.Count; i++)
            txn.Put(db, BitConverter.GetBytes(keysUnsorted[i]), BitConverter.GetBytes(i));

        using (var c = txn.CreateCursor(db))
        {
            int order = 0;
            while (c.MoveNext())
                Assert.AreEqual(keysSorted[order++], BitConverter.ToInt32(c.Current.Key, 0));
        }

Increase the mapsize on LightningEnvironment does not help. If I do not set the comparer function (i.e. commenting options.CompareWith...), it works with putting the items (but without ordering). Thanks.

[CoreyKaylor]

I've used your exact code provided here and ran it against netcoreapp1.0 and net451 and both pass. I'm not sure what's going on. Running on mono by chance?

[fangliuwh]

thanks for your reply. I tried again with .net 4.5.1, and use LightningDB 0.9.6.0 from nuget. I still have the exception, but different one as follows:

CallbackOnCollectedDelegate occurred Message: Managed Debugging Assistant 'CallbackOnCollectedDelegate' has detected a problem in 'ConsoleApplication1\ConsoleApplication1\bin\Debug\ConsoleApplication1.vshost.exe'. Additional information: A callback was made on a garbage collected delegate of type 'LightningDB!LightningDB.Native.CompareFunction::Invoke'. This may cause application crashes, corruption and data loss. When passing delegates to unmanaged code, they must be kept alive by the managed application until it is guaranteed that they will never be called.

Can you please try to increase the number of keys (e.g. from 10K to 100K, or 1Million)? Probably the GC behaves differently.

The complete code is:

class Program
{
    static void Main(string[] args)
    {
        new Tester().TransactionShouldSupportCustomComparer();
    }
}
public class Tester
{
    private string databaseName = "test";
    string envName = "testingLocalStore";

    public void TransactionShouldSupportCustomComparer()
    {
        Func<int, int, int> comparison = (l, r) => l.CompareTo(r);
        var options = new DatabaseConfiguration {Flags = DatabaseOpenFlags.Create};
        Func<byte[], byte[], int> compareWith =
            (l, r) => comparison(BitConverter.ToInt32(l, 0), BitConverter.ToInt32(r, 0));
        options.CompareWith(Comparer<byte[]>.Create(new Comparison<byte[]>(compareWith)));

        var _env = new LightningEnvironment(envName) {MapSize = (1L << 31)};
        _env.Open();

        using (var t1 = _env.BeginTransaction())
        using (var db1 = t1.OpenDatabase(configuration: options))
        {
            t1.DropDatabase(db1);
            t1.Commit();
        }

        var keysUnsorted = Enumerable.Range(1, 10000).Randomize().ToList();
        var keysSorted = keysUnsorted.ToArray();
        Array.Sort(keysSorted, new Comparison<int>(comparison));

        using (var t2 = _env.BeginTransaction())
        {
            var db = t2.OpenDatabase(configuration: options);
            for (var i = 0; i < keysUnsorted.Count; i++)
                t2.Put(db, BitConverter.GetBytes(keysUnsorted[i]), BitConverter.GetBytes(i));
            t2.Commit();
        }

        using (var t2 = _env.BeginTransaction())
        {
            var db = t2.OpenDatabase(configuration: options);
            using (var c = t2.CreateCursor(db))
            {
                int order = 0;
                while (c.MoveNext())
                    if(keysSorted[order++] != BitConverter.ToInt32(c.Current.Key, 0))
                        throw new InvalidOperationException("not sorted.");
            }
        }
    }
}
public static class IEnumerableExtensions
{

    public static IEnumerable<T> Randomize<T>(this IEnumerable<T> source)
    {
        var random = new Random();
        List<T> list = source.ToList<T>();
        for (int index1 = 0; index1 < list.Count*3; ++index1)
        {
            int index2 = random.Next(0, list.Count);
            int index3 = random.Next(0, list.Count);
            T obj = list[index2];
            list[index2] = list[index3];
            list[index3] = obj;
        }
        return (IEnumerable<T>) list;
    }
}

I looked into the decompiled code of DatabaseConfiguration in lightningDB to understand why and get the following:

public class DatabaseConfiguration
{
  private IComparer<byte[]> _comparer;
  private IComparer<byte[]> _duplicatesComparer;

  public DatabaseOpenFlags Flags { get; set; }

public DatabaseConfiguration()
{
  this.Flags = DatabaseOpenFlags.None;
}

internal void ConfigureDatabase(LightningTransaction tx, LightningDatabase db)
{
  if (this._comparer != null)
    Lmdb.mdb_set_compare(tx.Handle(), db.Handle(), new CompareFunction(this.Compare));
  if (this._duplicatesComparer == null)
    return;
  Lmdb.mdb_set_dupsort(tx.Handle(), db.Handle(), new CompareFunction(this.IsDuplicate));
}

private int Compare(ref ValueStructure left, ref ValueStructure right)
{
  return this._comparer.Compare(left.GetBytes(), right.GetBytes());
}

private int IsDuplicate(ref ValueStructure left, ref ValueStructure right)
{
  return this._duplicatesComparer.Compare(left.GetBytes(), right.GetBytes());
}

public void CompareWith(IComparer<byte[]> comparer)
{
  this._comparer = comparer;
}

public void FindDuplicatesWith(IComparer<byte[]> comparer)
{
  this._duplicatesComparer = comparer;
}

}

Somehow CompareFunction created in the following could be garbage collected?

Lmdb.mdb_set_compare(tx.Handle(), db.Handle(), new CompareFunction(this.Compare));

Thanks!

[CoreyKaylor]

Okay, bumping up the iterations and forcing garbage collection I was able to get an exception. For me it manifested as an AccessViolationException. I've got something local that fixes the problem and I should have a new nuget published today or tomorrow. I think in the end it wasn't being collected, it was being moved and therefore the reference in the native memory space wasn't pointing to the same thing anymore. More reading on the UnmanagedFunctionPointer indicates that the delegate is only pinned long enough for a single callback, or rather it's only guaranteed for the single callback in the scope of you calling the native function. Hopefully this fixes your issue too, if not I'll go back to the drawing board.

[CoreyKaylor]

0.9.7 has the fix, feel free to reopen if you still have problems

[fangliuwh]

indeed it works. Many thanks!