Please support a modern very high performance version of System.IO.Log

[AceHack]

I'm looking for an very high performance version Algorithms for Recovery and Isolation Exploiting Semantics (ARIES) like .net standard/core namspace. It does not need to be exactly ARIES but would need to be sufficent to be the start of a number of different data oriented projects that choose .NET. I would use this and I think others would use this to start a new eco system of databases, no sql, messaging, streaming and data oriented .NET core open source projects.

[AceHack]

Basically what I'm looking for is the best Write Ahead Log (WAL) possible on a cross platform language.

[karelz]

Your request seems to be reasonable building component for database-like systems. However, it does not seem to fit the scope of CoreFX/BCL and would be IMO better served as independent library.

In general we consider extending CoreFX with APIs which:

1. Extend existing APIs/libraries
2. Expose low-level runtime capabilities (e.g. Threading)
3. Would be used or exposed by other BCL/CoreFX libraries

See dotnet/runtime#22228 for examples of APIs we consider out of scope of CoreFX.

[AceHack]

There are a few reasons I think this belongs to CoreFX/BCL instead of a library. 1) It was part of BCL of full .NET framework on windows (This is not a very good reason) 2) It requires low level below FileStream access to things like overlapped unbuffered IO, I/O completion ports or maybe just the native win32 apis for CLFS (Common Log File System) and on Linux would require below filestream access as well for best performace and true durable compliance with things like write though and no caching so you can be sure things are commited and durable and not in buffers either hardward or software. Also flushing occasionally is not an option because it's so SLOW to operate in that mannor. 3) To really have the base needed for the performance, quality, and durability guarantees having a high quality team like MS be the curators would be wonderful. 4) If SQL server would be willing to share some techniques (maybe not) that could provide some latest techniques for high quality transaction log.

Thanks.

[AceHack]

Here is some example low level code that would likely be used on windows. This is not a full WAL or ARIES yet but I image it would use a lot of the same constructs. You can see how many low level OS constructs are required and one reason why I think it would be best to be in CoreFX.

This code is to allow for fast sequential writes to a file while making sure every write is "commited" and durable all the way though all buffers including hardware without the need for constant flushing. It also does a lot of work to create page alligned memory buffers i.e. the virtual alloc buffer and figure the sector size of the disk. All of these things are needed for optimal write performance.

Without a better abstraction layer for library writers like mono provided writing new cross platform libraries that need to get into the guts of the different OSs involved is much harder with CoreFX. See https://github.com/dotnet/coreclr/issues/930 for examples of the kind of abstractions I'm talking about.

Also if somehow FileStream is much better in .NET core than full .NET and will allow fast sequential writes while making sure there are no buffers in between all the way to the hardware, page alligned writes, figuring out sector size, then I would love to understand better. Thanks.

Here is some more info on why page and sector alignment are so important. http://programmingaddicted.blogspot.com/2011/05/unbuffered-overlapped-io-in-net.html https://msdn.microsoft.com/en-us/library/windows/desktop/cc644950%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396 https://arxiv.org/ftp/cs/papers/0502/0502012.pdf http://www.installsetupconfig.com/win32programming/windowsfileapis4_13.html

```c# [SecuritySafeCritical] public sealed class UnbufferedFile : IDisposable { [SecurityCritical] private readonly SafeFileHandle _safeFileHandle; private readonly int _sectorSize; private long _fileSize; private volatile bool _disposed; public SafeFileHandle SafeHandle { [SecurityCritical] [SecurityPermission(SecurityAction.InheritanceDemand, Flags = SecurityPermissionFlag.UnmanagedCode)] get { return _safeFileHandle; } } public int SectorSize { get { return _sectorSize; } } [SecuritySafeCritical] public UnbufferedFile(string path) { _sectorSize = GetSectorSize(path); _safeFileHandle = WinAPI.CreateFile(path, FileAccess.ReadWrite, FileShare.None, IntPtr.Zero, FileMode.OpenOrCreate, FileOptions.WriteThrough | FileOptions.Asynchronous | WinAPI.NoBuffering, IntPtr.Zero); if (_safeFileHandle.IsInvalid) throw new Win32Exception(); bool successfulBind = false; try { _fileSize = GetFileSize(); successfulBind = ThreadPool.BindHandle(_safeFileHandle); } finally { if (!successfulBind) _safeFileHandle.Close(); } if (!successfulBind) throw new IOException("Binding to handle failed."); } [SecuritySafeCritical] public void Dispose() { _disposed = true; if (_safeFileHandle != null) _safeFileHandle.Dispose(); } public long Length { [SecuritySafeCritical] get { if (_disposed) throw new ObjectDisposedException("UnbufferedFile", "UnbufferedFile already disposed."); return _fileSize; } [SecuritySafeCritical] set { if (_disposed) throw new ObjectDisposedException("UnbufferedFile", "UnbufferedFile already disposed."); if (value < 0L) throw new ArgumentOutOfRangeException("value", "File size must be positive"); if (!WinAPI.SetFilePointerEx(_safeFileHandle, value, IntPtr.Zero, SeekOrigin.Begin)) throw new Win32Exception(); if (!WinAPI.SetEndOfFile(_safeFileHandle)) throw new Win32Exception(); _fileSize = value; } } [SecuritySafeCritical] public Task ReadAsync(long position, VirtualAllocBuffer buffer, long offset, int count) { return ReadAsync(position, buffer, offset, count, new CancellationToken()); } [SecuritySafeCritical] public Task ReadAsync(long position, VirtualAllocBuffer buffer, long offset, int count, CancellationToken cancellationToken) { EnsureReadWriteParameters(position, buffer, offset, count); return ReadWriteCoreAsync(_safeFileHandle, position, buffer, offset, count, true, cancellationToken); } [SecuritySafeCritical] public Task WriteAsync(long position, VirtualAllocBuffer buffer, long offset, int count) { return WriteAsync(position, buffer, offset, count, new CancellationToken()); } [SecuritySafeCritical] public Task WriteAsync(long position, VirtualAllocBuffer buffer, long offset, int count, CancellationToken cancellationToken) { EnsureReadWriteParameters(position, buffer, offset, count); return ReadWriteCoreAsync(_safeFileHandle, position, buffer, offset, count, false, cancellationToken); } [SecurityCritical] private static int GetSectorSize(string path) { string rootPath = Path.GetPathRoot(path); if (string.IsNullOrEmpty(rootPath)) rootPath = Path.GetPathRoot(Environment.CurrentDirectory); if (string.IsNullOrEmpty(rootPath)) throw new InvalidOperationException("Can't get root path"); uint size; uint i; WinAPI.GetDiskFreeSpace(rootPath, out i, out size, out i, out i); return (int)size; } [SecurityCritical] private long GetFileSize() { int highSize; var lowSize = WinAPI.GetFileSize(_safeFileHandle, out highSize); if (lowSize == -1 && Marshal.GetLastWin32Error() != 0) throw new Win32Exception(); return (long)highSize << 32 | (uint)lowSize; } [SecurityCritical] private unsafe static Task ReadWriteCoreAsync(SafeFileHandle fileHandle, long position, VirtualAllocBuffer buffer, long offset, int count, bool isRead, CancellationToken cancellationToken) { var memoryStartLocation = new IntPtr(unchecked((long)((ulong)buffer.SafeHandle.DangerousGetHandle() + (ulong)offset))); var taskCompletionSource = new TaskCompletionSource

[karelz]

Here's my view on your reasons to add the library into CoreFX:

1. System.IO.Log was part of .NET Framework
   • This is a good argument for porting the code over "as is", but it does not require making the code very high-performance implementation. If there is enough upvotes, we can definitely consider bringing it into future versions of .NET Framework Compatibility Pack - see dotnet/runtime#23974.
2. It requires low-level file I/O
   • Any library can make low-level OS API calls. That by itself does not mean it has to be part of CoreFX.
3. Microsoft ownership guarantees high quality
   • This is common (mostly hidden) reason of asks to add APIs into CoreFX repo (and other MS repos). Obviously, it would not scale if we said yes to every such addition - CoreFX would become the kitchen sink repo of everything low-level enough. That's why we have to prioritize based on customer demand (e.g. upvotes) and make sure added libraries are in line with the rest of CoreFX repo.
   • Also note that there are PLENTY of high quality libraries maintained by other companies or community - see e.g. list of .NET Foundation projects or most popular NuGet libraries. Delivering a high quality library is hard and lots of work, but there are MANY communities and companies, not just Microsoft, capable of delivering such value.
4. SQL Server team could contribute
   • I can't speak on SQL Server organization behalf. It depends on their business priorities. Note that their ability to contribute should not be limited to only Microsoft-owned repos.

I would recommend to split this issue into 2 separate items:

• CoreFX issue to port System.IO.Log from .NET Framework as is for compatibility reasons. We would monitor demand via upvotes / replies.
• CoreFX-independent effort to build high-performance ARIES library, probably with database folks involvement. It might be good to check if there were similar efforts in the past / in progress.

[sakno]

I'm voting for Write Ahead Log as a part of .NET because I have implementation of Raft Consensus Algorithm for ASP.NET Core which requires to have persistent replication log. At this moment the log is represented by interface and its actual implementation is delegated to the consumer.

[danmoseley]

@davidfowl what are thoughts from ASPNET perspective

[sakno]

I would like to share implementation of Write Ahead Log that is suitable for general purpose use as a proof of concept. Sources are here and licensed under MIT. What was done:

• Search, drop, append, commit core operations for manipulations with log entries
• Implementation is fully asynchronous
• Reduced memory allocations: they are replaced by renting where possible
• Thread safety
• Parallel reads
• Consumer able to define its own binary format for log records
• Configurable caches for fast lookup
• Log compaction using snapshotting approach
• Logical partitioning
• O(1) random access to log records The implementation relies only on existing .NET Standard API so it is portable across OSes and .NET runtimes.

What was not done (and probably won't):

• Low-level optimizations that was mentioned by @AceHack because of inability to port them across different platforms

Originally, it was developed for log replication as described by Raft consensus algorithm so it has some specific API such as Raft node state. However, you can just ignore this part of API. I can say that development of fully general-purpose WAL is not so easy because underlying database engine may require some specific features. Also, I don't have benchmarks because various C# implementations of oss databases have very specific log engines so their performance are not comparable.

[hiteshmadan]

@AceHack @sakno Please checkout FasterLog from here: https://github.com/microsoft/FASTER

FASTER is a really fast cross-platform embedded key-value store, and its' internal log implementation was upgraded into a public API in the last few weeks. Super impressive performance characteristics imho - try out the FasterLogSample csproj on a machine with an NVMe SSD and hopefully you'll be impressed with what you see!

cc @badrishc

(disclaimer - I've been using it for a work project and actively collaborating with the main author)

[AceHack]

@hiteshmadan This seems pretty amazing, thanks for pointing this out.

[danmoseley]

@hiteshmadan you might consider a PR to add it to this.

https://github.com/microsoft/dotnet/blob/master/dotnet-developer-projects.md

[badrishc]

Submitted a PR as suggested, here. To expand a bit, FasterLog (docs) is a latch-free concurrent persistent log library for .NET. It supports concurrent appends, group commit, multiple persistent iterators (each of which may itself be concurrently used), random reads, and log truncation from head. It can run over sharded and tiered storage backend devices (we have out-of-the-box device implementations for local storage and Azure Page Blobs). Our benchmarks are able to easily saturate 2+ GB/sec on a single local NVMe SSD, and 4+ GB/sec when run over two (sharded) SSDs.

[stephentoub]

Closing as being addressed by an existing library in the ecosystem.