search

dotnet / dotNext

Next generation API for .NET
https://dotnet.github.io/dotNext/
MIT License
1.56k stars 119 forks source link

DotNext.Net.Cluster crash in production since I think version 5.4.0 #242

Closed guillaume-chervet closed 1 week ago

guillaume-chervet commented 3 weeks ago

hi @sakno,

We have crashed in production that lock nodes. The slimfaas code did not change things like to This part, we only update libraries : https://github.com/AxaFrance/SlimFaas/commit/b26e3bdb8a05bbd41140e24a34ebc9598f799661 I'am not sure but I think it is link to these changes :

DotNext.Net.Cluster 5.4.0

Changed binary file format for WAL for more efficient I/O. A new format is incompatible with all previous versions. To enable legacy format, set PersistentState.Options.UseLegacyBinaryFormat property to true Introduced a new experimental binary format for WAL based on sparse files. Can be enabled with PersistentState.Options.MaxLogEntrySize property

We took the new default system and we have this new error that happen sometime and crash the node 👍 

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'fileOffset')
         at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
         at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
         at System.IO.RandomAccess.WriteAsync(SafeFileHandle, IReadOnlyList`1, Int64, CancellationToken) + 0x3d
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.<WriteThroughAsync>d__17.MoveNext() + 0x2ad
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendCachedAsync>d__44`1.MoveNext() + 0x740
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAsync>d__47`1.MoveNext() + 0x4f2
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAndCommitSlowAsync>d__50`1.MoveNext() + 0x164
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x29
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<AppendEntriesAsync>d__101`1.MoveNext() + 0x82b
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x21
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x5ac
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x7f7
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<<Invoke>g__Awaited|10_0>d.MoveNext() + 0xac
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'fileOffset')
         at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
         at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
         at System.IO.RandomAccess.WriteAsync(SafeFileHandle, IReadOnlyList`1, Int64, CancellationToken) + 0x3d
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.<WriteThroughAsync>d__17.MoveNext() + 0x2ad
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendCachedAsync>d__44`1.MoveNext() + 0x740
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAsync>d__47`1.MoveNext() + 0x4f2
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAndCommitSlowAsync>d__50`1.MoveNext() + 0x164
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x29
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<AppendEntriesAsync>d__101`1.MoveNext() + 0x82b
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x21
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x5ac
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x7f7
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<<Invoke>g__Awaited|10_0>d.MoveNext() + 0xac
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'fileOffset')
         at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
         at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
         at System.IO.RandomAccess.WriteAsync(SafeFileHandle, IReadOnlyList`1, Int64, CancellationToken) + 0x3d
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.<WriteThroughAsync>d__17.MoveNext() + 0x2ad
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendUncachedAsync>d__45`1.MoveNext() + 0x214
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAsync>d__47`1.MoveNext() + 0x2e5
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAndCommitSlowAsync>d__50`1.MoveNext() + 0x164
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x29
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<AppendEntriesAsync>d__101`1.MoveNext() + 0x82b
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x21
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x5ac
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x7f7
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<<Invoke>g__Awaited|10_0>d.MoveNext() + 0xac
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'fileOffset')
         at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
         at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
         at System.IO.RandomAccess.WriteAsync(SafeFileHandle, IReadOnlyList`1, Int64, CancellationToken) + 0x3d
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.<WriteThroughAsync>d__17.MoveNext() + 0x2ad
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendUncachedAsync>d__45`1.MoveNext() + 0x214
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAsync>d__47`1.MoveNext() + 0x2e5
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAndCommitSlowAsync>d__50`1.MoveNext() + 0x164
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x29
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<AppendEntriesAsync>d__101`1.MoveNext() + 0x82b
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x21
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x5ac
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x7f7
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<<Invoke>g__Awaited|10_0>d.MoveNext() + 0xac
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'fileOffset')
         at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
         at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
         at System.IO.RandomAccess.WriteAsync(SafeFileHandle, IReadOnlyList`1, Int64, CancellationToken) + 0x3d
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.<WriteThroughAsync>d__17.MoveNext() + 0x2ad
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendUncachedAsync>d__45`1.MoveNext() + 0x214
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAsync>d__47`1.MoveNext() + 0x2e5
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAndCommitSlowAsync>d__50`1.MoveNext() + 0x164
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x29
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<AppendEntriesAsync>d__101`1.MoveNext() + 0x82b
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x21
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x5ac
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x7f7
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<<Invoke>g__Awaited|10_0>d.MoveNext() + 0xac
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'fileOffset')
         at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
         at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
         at System.IO.RandomAccess.WriteAsync(SafeFileHandle, IReadOnlyList`1, Int64, CancellationToken) + 0x3d
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.<WriteThroughAsync>d__17.MoveNext() + 0x2ad
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) + 0x22
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendUncachedAsync>d__45`1.MoveNext() + 0x214
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAsync>d__47`1.MoveNext() + 0x2e5
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
         at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<AppendAndCommitSlowAsync>d__50`1.MoveNext() + 0x164
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x29
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<AppendEntriesAsync>d__101`1.MoveNext() + 0x82b
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult() + 0x16
         at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16) + 0x2d
         at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16) + 0x21
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x5ac
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<AppendEntriesAsync>d__79.MoveNext() + 0x7f7
      --- End of stack trace from previous location ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e

or like this 

at System.ThrowHelper.ThrowArgumentOutOfRangeException_NeedNonNegNum(String) + 0x30
at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.Initialize() + 0x2a2
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<.ctor>g__CreateTables|28_1(SortedSet`1, DirectoryInfo, Int32, Int32, PersistentState.BufferManager&, Int32, PersistentState.WriteMode, Int64) + 0x14f
at DotNext.Net.Cluster.Consensus.Raft.PersistentState..ctor(DirectoryInfo, Int32, PersistentState.Options) + 0x445
at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine..ctor(DirectoryInfo, Int32, MemoryBasedStateMachine.Options) + 0x57
at SlimData.SlimPersistentState..ctor(String path) + 0x10c
at SlimFaas!<BaseAddress>+0xe59cb0
at System.Reflection.DynamicInvokeInfo.InvokeWithFewArguments(IntPtr, Byte&, Byte&, Object[], BinderBundle, Boolean) + 0x84
at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0xf3
at Internal.Reflection.Execution.MethodInvokers.InstanceMethodInvoker.CreateInstance(Object[], BinderBundle, Boolean) + 0x43
at Internal.Reflection.Core.Execution.MethodBaseInvoker.CreateInstance(Object[], Binder, BindingFlags, CultureInfo) + 0x3e
at System.Reflection.Runtime.MethodInfos.RuntimePlainConstructorInfo`1.Invoke(BindingFlags, Binder, Object[], CultureInfo) + 0x5b
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext) + 0x74
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument) + 0xa9
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite, ServiceProviderEngineScope) + 0x1d
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier) + 0xf7
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xf7
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier, ServiceProviderEngineScope) + 0x39
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type) + 0x2d
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type) + 0x51
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) + 0x2f
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite, RuntimeResolverContext) + 0xf
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext) + 0x74
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument) + 0xa9
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext) + 0x86
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext) + 0x74
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument) + 0xa9
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite, ServiceProviderEngineScope) + 0x1d
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier) + 0xf7
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xf7
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier, ServiceProviderEngineScope) + 0x39
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type) + 0xf
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type) + 0x51
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) + 0x2f
at DotNext.Net.Cluster.Consensus.Raft.Http.ConfigurationExtensions.UseConsensusProtocolHandler(IApplicationBuilder) + 0x5d
at SlimData.Startup.Configure(IApplicationBuilder app) + 0x15
at Program.<Main>$(String[] args) + 0x10f5
sakno commented 3 weeks ago

Have you set UseLegacyBinaryFormat property to true?

guillaume-chervet commented 3 weeks ago

Yes, It is working when I set it back to the old mode. I always errase the previous data storage when I test. @sakno

sakno commented 3 weeks ago

Do you mean that it crashes on empty WAL with a new format?

guillaume-chervet commented 3 weeks ago

It happen with empty WAL sometime and sometime after an amount of time with the existing WAL @sakno .

sakno commented 3 weeks ago

Do you have a stable repro? I see that the second stack trace is from the tests in your repository.

guillaume-chervet commented 3 weeks ago

It is a kind of random behavior. But when it start to happen it does not stop.

First logs comes from our production. Second comes from dev environment from one of 3 nodes at startup. @sakno

sakno commented 3 weeks ago

It could happen if you trying to open WAL produced by version < 5.4.0 with a new version >=5.4.0 without UseLegacyBinaryFormat set to true. Are you sure that dev environment starts with clean environment without older WAL files?

guillaume-chervet commented 3 weeks ago

Yes I am sure. My store for testing was completly errased. Idem for the updated production. @sakno

guillaume-chervet commented 3 weeks ago

SlimFaas is compiled in AOT.

sakno commented 3 weeks ago

The second stack trace indicates that WAL is trying to read existing files:

at System.IO.RandomAccess.ValidateInput(SafeFileHandle, Int64, Boolean) + 0x5f
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.Table.Initialize() + 0x2a2
at DotNext.Net.Cluster.Consensus.Raft.PersistentState.<.ctor>g__CreateTables|28_1(SortedSet`1, DirectoryInfo, Int32, Int32, PersistentState.BufferManager&, Int32, PersistentState.WriteMode, Int64) + 0x14f

There is a code for Initialize: https://github.com/dotnet/dotNext/blob/cacf3e573b460469786314428617c4ce43387194/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.Partition.cs#L507-L552

To get an exception like in your stack trace the program needs to go to the second or third if branch. It is possible only if there is a file in the file system.

guillaume-chervet commented 3 weeks ago

forgot the latest logs @sakno I may made a mistake in our dev kubernetes environment.

Here the logs my collegues sent to me from the crash in production. Occur with the new protocol (in random laps of time near 48 hours and do not happen with the old one). I think it manage near 400 000 writes operation by day. slimfaas-1-slimfaas.log slimfaas-2-slimfaas.log slimfaas-0-slimfaas.log

I do no kown where can come from the negative number.

sakno commented 3 weeks ago

How WAL is configured? How many records per partition, parallel IO, etc? What's the target architecture, x86_64?

guillaume-chervet commented 3 weeks ago

Target architecture is x86 64. The other options I do not know what it is. Here is the SlimData persistent constructor https://github.com/AxaFrance/SlimFaas/blob/2ca3a8c7589b87dcd560164d7ed643f8f17aa89b/src/SlimData/SlimPersistentState.cs#L19

Thank you @sakno for your help

sakno commented 3 weeks ago

It's hard to say what's the root cause of the problem because there is no stable repro. I can only guess. Possibly it happens because of network timeouts leading to cancellation of the token used by WAL internally to perform I/O. Some I/O were done in a way not safe for cancellation, I've prepared the potential fix. I can't release it right now.

sakno commented 3 weeks ago

Did you have a chance to check the fix?

guillaume-chervet commented 3 weeks ago

Hi @sakno do you have a way to publish an alpha?

guillaume-chervet commented 3 weeks ago

My level in c# is not the best 😜 It my favorite one but i do not code a lot with (unfortunately).

sakno commented 2 weeks ago

You can reference a project explicitly from your csproj file without published alpha.

sakno commented 1 week ago

Release 5.7.0 has been published.

guillaume-chervet commented 1 week ago

Thank you @sakno I test it today and tell you if it fix the problem