Giorgi / DuckDB.NET

Bindings and ADO.NET Provider for DuckDB
https://duckdb.net
MIT License
338 stars 61 forks source link

Making use of `LibraryImport` to reduce marshalling, smaller optimizations #194

Open kirides opened 3 weeks ago

kirides commented 3 weeks ago

This project may benefit a lot from using LibraryImport-Attribute instead of DllImport (or combined with? idk if and how fallback would work)

As DuckDb is in Process, reducing the marshalling overhead by introducing LibraryImport may further increase throughput.

Side note: Things like StringVectorDataReader may also benefit from using string.Create(....) to reduce allocations to just a single "string", removing the additional new char[...] allocation.

Also DuckDB.NET.Data.Extensions.GuidConverter may benefit a lot from using Span<char> buffer = stackalloc char[36] to build the guid-representation in memory without allocating at all.

private static readonly char[] HexDigits = "0123456789abcdef".ToCharArray();
public static Guid ConvertToGuid_Optimized(DuckDB.NET.Native.DuckDBHugeInt input)
{
    Span<char> buffer = stackalloc char[32];
    long num = input.Upper ^ long.MinValue;
    int position = 0;
    ByteToHex(buffer, ref position, (ulong)((num >> 56) & 0xFF));
    ByteToHex(buffer, ref position, (ulong)((num >> 48) & 0xFF));
    ByteToHex(buffer, ref position, (ulong)((num >> 40) & 0xFF));
    ByteToHex(buffer, ref position, (ulong)((num >> 32) & 0xFF));
    //buffer[position++] = '-';
    ByteToHex(buffer, ref position, (ulong)((num >> 24) & 0xFF));
    ByteToHex(buffer, ref position, (ulong)((num >> 16) & 0xFF));
    //buffer[position++] = '-';
    ByteToHex(buffer, ref position, (ulong)((num >> 8) & 0xFF));
    ByteToHex(buffer, ref position, (ulong)(num & 0xFF));
    //buffer[position++] = '-';
    ByteToHex(buffer, ref position, (input.Lower >> 56) & 0xFF);
    ByteToHex(buffer, ref position, (input.Lower >> 48) & 0xFF);
    //buffer[position++] = '-';
    ByteToHex(buffer, ref position, (input.Lower >> 40) & 0xFF);
    ByteToHex(buffer, ref position, (input.Lower >> 32) & 0xFF);
    ByteToHex(buffer, ref position, (input.Lower >> 24) & 0xFF);
    ByteToHex(buffer, ref position, (input.Lower >> 16) & 0xFF);
    ByteToHex(buffer, ref position, (input.Lower >> 8) & 0xFF);
    ByteToHex(buffer, ref position, input.Lower & 0xFF);
    return Guid.ParseExact(buffer, "n");

    static void ByteToHex(Span<char> buffer, ref int position, ulong value)
    {
        buffer[position++] = HexDigits[(value >> 4) & 0xF];
        buffer[position++] = HexDigits[value & 0xF];
    }
}
Giorgi commented 3 weeks ago

I have been planning to look into LibraryImport for quite some time but didn't get to it. Hopefully I will find time for it in the next few weeks.

Other suggestions look interesting too. If you want to feel free to send a PR and I will be happy to review it.

Giorgi commented 2 weeks ago

@kirides Implemented Guid and BitString suggestions in c83fafaaeb

Giorgi commented 2 weeks ago

LibraryImport requires #ifdef-s to check for .net 7 as it is unavailable in the previous versions. @kirides Want to send a PR?