Spreads / Spreads.LMDB

Low-level zero-overhead and the fastest LMDB .NET wrapper with some additional native methods useful for Spreads
http://docs.dataspreads.io/spreads/libs/lmdb/api/README.html
Mozilla Public License 2.0
80 stars 9 forks source link

Random AccessViolationException #9

Closed jakoss closed 6 years ago

jakoss commented 6 years ago

On following code:

public unsafe void PutSubFingerprint(SubFingerprintDataDTO subFingerprintDataDTO)
        {
            var subFingerprintKey = new DirectBuffer(BitConverter.GetBytes(subFingerprintDataDTO.SubFingerprintReference));
            var value = ZeroFormatterSerializer.Serialize(subFingerprintDataDTO);
            fixed (byte* array = value)
            {
                var directBuffer = new DirectBuffer(value.Length, array);
                databasesHolder.SubFingerprintsDatabase.Put(tx, ref subFingerprintKey, ref directBuffer);
            }
        }

I get randomly occuring AccessViolationException on line databasesHolder.SubFingerprintsDatabase.Put(tx, ref subFingerprintKey, ref directBuffer);.

I'm calling this code around 2000 times in a single Transaction (i don't know if that have any meaning in this context

By random i mean i can't find reproduction pattern. I'm building database using the same data in the same order. Sometimes it works and sometime it fails. When it fails it's always on another entry. So i don't think it's data fault. Am i doing something wrong?

buybackoff commented 6 years ago

subFingerprintKey is not fixed

buybackoff commented 6 years ago

I get randomly occuring

Random errors in unsafe context with native interop most likely (almost always) mean that something is not pinned and GC is relocating objects. DirectBuffer doesn't pin for you.

jakoss commented 6 years ago

Ok, that makes sense. And if i'm using generic Put/Get methods do i need to pin memory too?

buybackoff commented 6 years ago

I have this in Readme:

For writes, the memory behind DirectBuffer MUST BE pinned.

Probably should make it bold in addition to all-caps cursive :)

And if i'm using generic Put/Get methods do i need to pin memory too?

No, they are on stack and pointer to them is fixed for the method execution lifetime using Unsafe.AsPointer. Generics only work for blittable types and currently you must opt-in for custom structs as described in the readme section. Primitive types work out of the box.

jakoss commented 6 years ago

It's a shame that you cannot force pinning explicitly using C#. But i think using this library require little bit more sophisticated C# knowledge anyway.

Pinning every value under DirectBuffer helped. Thanks a lot!

buybackoff commented 6 years ago

using(Memory<byte>.Pin()){} could be more handy than fixed statement.

It's a shame that you cannot force pinning explicitly using C#.

Pinning has some costs, but unavoidable for native interop. All the tools for pinning are there: fixed, Memory<byte>.Pin(), GCHandle.Alloc(..pinned), etc.

Span, Memory & Co use refs but GC cannot know when a ref is used by a native method.