UnauthorizedAccessException thrown when accessing http raft node state

[cdavernas]

What happens?

An UnauthorizedAccessException is thrown when restarting an http raft node with persistence enabled.

What is expected?

The http raft node starts without throwing exception

Configuration:

Here's the relevent content of my appsettings.json file:

{
  "Partitioning": false,
  "LowerElectionTimeout": 150,
  "UpperElectionTimeout": 300,
  "RequestTimeout": "00:10:00",
  "ColdStart": false,
  "RequestJournal": {
    "MemoryLimit": 5,
    "Expiration":  "00:01:00"
  }
}

Here's my Program.cs file:

var builder = WebApplication.CreateBuilder(args);

var port = int.Parse(args[0]);
if (args.Length > 1)
    builder.Configuration["ContactNode"] = $"http://localhost:{int.Parse(args[1])}";
builder.Configuration["PublicEndPoint"] = $"http://localhost:{port}";
builder.Configuration["LocalNode"] = $"http://localhost:{port}";

builder.WebHost.ConfigureKestrel(kestrel =>
{
    kestrel.ListenLocalhost(port);
});

builder.Services.UsePersistentConfigurationStorage(Path.Combine(port.ToString(), "config")); 
builder.Services.UsePersistenceEngine<IClusterStateMachine, ClusterStateMachine>();
builder.Services.AddSingleton<IPeerLifetime, PeerLifetime>();

builder.Services.Configure<ClusterStateMachineOptions>(options => Path.Combine(port.ToString(), "state"));

builder.JoinCluster();
builder.JoinMesh();

var app = builder.Build();

if (!app.Environment.IsDevelopment())
    app.UseExceptionHandler("/error");
app.UseStaticFiles();
app.UseConsensusProtocolHandler();
app.UseHyParViewProtocolHandler();
app.UseRouting();

---

Any idea of why I'm facing this issue? Thanks in advance and big up for your amazing project!

[sakno]

@cdavernas , do you have stack trace details? The code of the library doesn't throw this type of exception. Probably, your process doesn't have enough privileges to use particular port or file path.

[cdavernas]

@sakno I'll provide the traces tomorrow (in bed right now), but it's not related to hosting, it's an IO issue. At first launch, the state folder is created and works as expected. Restarting the app without removing said folder throws the exception after attempting to write to the 0 file. Now that I think of it, it might occur because the faulted node is systematically starting with the boostrap flag set to true, but I need to verify that theory.

[cdavernas]

My theory appears to be wrong. It occurs both when coldStart is set to true and to false, unhappilly.

Here's the stacktrace:

   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.RandomAccess.Write(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Partition.Initialize()
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState..ctor(DirectoryInfo path, Int32 recordsPerPartition, Options configuration)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine..ctor(DirectoryInfo path, Int32 recordsPerPartition, Options configuration)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine..ctor(String path, Int32 recordsPerPartition, Options configuration)
   at Test.Application.Services.ClusterStateMachine..ctor(ILogger`1 logger, IOptions`1 options, ISerializer serializer) in [RedactedPath]\Test\Services\ClusterStateMachine.cs:line 28
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at DotNext.Net.Cluster.Consensus.Raft.Http.ConfigurationExtensions.UseConsensusProtocolHandler(IApplicationBuilder builder)
   at Program.<<Main>$>d__0.MoveNext() in    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.RandomAccess.Write(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Partition.Initialize()
   at DotNext.Net.Cluster.Consensus.Raft.PersistentState..ctor(DirectoryInfo path, Int32 recordsPerPartition, Options configuration)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine..ctor(DirectoryInfo path, Int32 recordsPerPartition, Options configuration)
   at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine..ctor(String path, Int32 recordsPerPartition, Options configuration)
   at EventMesh.Application.Services.ClusterStateMachine..ctor(ILogger`1 logger, IOptions`1 options, ISerializer serializer) in [RedactedPath]Test\src\core\Test.Application\Test.Application\Services\ClusterStateMachine.cs:line 28
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at DotNext.Net.Cluster.Consensus.Raft.Http.ConfigurationExtensions.UseConsensusProtocolHandler(IApplicationBuilder builder)
   at Program.<<Main>$>d__0.MoveNext() in [RedactedPath]\Test\Program.cs:line 43

Edit:

It only seems to occur when the 0 file is present. Restarting a node that does not yet own its own copy works as expected.

[sakno]

Could you please post exact message from exception? Probably the program trying to access the same file.

[cdavernas]

The exact message is:

Access to the path '[RedactedPath]\test\bin\Debug\net6.0\5125\0' is denied.

No inner exception, no data and no additional info, aside from the stack trace supplied in my previous comment

[sakno]

Seems like another process trying to access the same file. Each node must have its own Write-Ahead Log.

[cdavernas]

Each node must have its own Write-Ahead Log.

That's the case though! I use the name of the node public endpoint's port as path to the state directory (i.e. 5125).

I really dont get it, cant make sense of it. If I set the breakpoint on the MemoryBasedStateMachine's constructor, I can delete the files, no problem, but just after stepping over, I get the exception.

[sakno]

Check who's else have access to the file. The project has example called RaftNode where three nodes use individual folders to store WAL. Moreover, the project is used in production. I see some problems with the local environment, not with code base.

[cdavernas]

Yeah, your test project works as expected. Im probably doing something stupid that I cant seem to figure out. Ill share a repo for reproduction purpose.

[sakno]

Anyway, you should not use HyParView for member discovery especially in combination with Raft. HyParView is a membership protocol for Gossip-based messaging with weak consistency of cluster in contrast to Raft membership protocol with strong consistency. HyParView and Raft are orthogonal things for different purposes.