search

datalust / seq-tickets

Issues, design discussions and feature roadmap for the Seq log server
https://datalust.co/seq
97 stars 5 forks source link

SEQ docker instance fails to start after running fine for months #2039

Closed HakanL closed 11 months ago

HakanL commented 11 months ago

Describe the bug After a clean shut down my instance fails to start. I'm getting this error in the docker/seq log:

 ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.
   caused by: snappy: error snappy reading buffered frame header
   caused by: snappy: corrupt input (unsupported chunk length: 107)

Environment (please complete the following information):

Additional context Here's a part of the log file:

{"@t":"2023-12-06T14:35:32.8744854Z","@mt":"Storage optimization took {Elapsed} ms; total iteration time was {Iteration} ms","Elapsed":897,"Iteration":915.2083}
{"@t":"2023-12-06T14:40:28.4613835Z","@mt":"Metrics sampled","SystemMemoryUtilization":0.504524333577892,"ProcessWorkingSetBytes":251408384,"ProcessCpuPercentage":0.12333318357347471,"GCTotalHeapSizeBytes":0,"GCTotalFragmentedBytes":0,"GCPauseTimePercentage":0,"ProcessThreads":27,"ThreadPoolActiveThreads":4,"ThreadPoolAvailableWorkerThreads":32766,"ThreadPoolAvailableIOThreads":1000,"ArrivedEventsPerMinute":28,"IngestedEventsPerMinute":28,"IngestedBytesPerMinute":19792,"InvalidPayloadsPerMinute":0,"UnauthorizedPayloadsPerMinute":0,"TotalWriteTimeLastMinute":96,"BroadcastQueueCount":1,"BroadcastSubscriptionQueuesAggregateCount":0,"AspNetCoreHostingCurrentRequests":0,"AspNetCoreHostingFailedRequests":2,"AspNetCoreHostingRequestsPerSecond":37,"KestrelCurrentConnections":1,"KestrelConnectionsPerSecond":0,"KestrelConnectionQueueLength":0,"KestrelRequestQueueLength":0,"KestrelFailedTlsHandshakes":0}
{"@t":"2023-12-06T14:45:28.4574080Z","@mt":"Metrics sampled","SystemMemoryUtilization":0.5095981229862631,"ProcessWorkingSetBytes":251408384,"ProcessCpuPercentage":0.12333501014678853,"GCTotalHeapSizeBytes":0,"GCTotalFragmentedBytes":0,"GCPauseTimePercentage":0,"ProcessThreads":26,"ThreadPoolActiveThreads":3,"ThreadPoolAvailableWorkerThreads":32766,"ThreadPoolAvailableIOThreads":1000,"ArrivedEventsPerMinute":16,"IngestedEventsPerMinute":16,"IngestedBytesPerMinute":11213,"InvalidPayloadsPerMinute":0,"UnauthorizedPayloadsPerMinute":0,"TotalWriteTimeLastMinute":108,"BroadcastQueueCount":0,"BroadcastSubscriptionQueuesAggregateCount":0,"AspNetCoreHostingCurrentRequests":0,"AspNetCoreHostingFailedRequests":2,"AspNetCoreHostingRequestsPerSecond":44,"KestrelCurrentConnections":1,"KestrelConnectionsPerSecond":0,"KestrelConnectionQueueLength":0,"KestrelRequestQueueLength":0,"KestrelFailedTlsHandshakes":0}
{"@t":"2023-12-06T14:45:29.7747381Z","@mt":"Indexer starting with {Allocated:0.000} ms allocated","@r":["600000.000"],"Allocated":600000,"SourceContext":"Seq.Server.Features.Indexing.EventStoreIndexer"}
{"@t":"2023-12-06T14:45:29.7776440Z","@mt":"Computed {IndexDefinitionCount} index definitions","IndexDefinitionCount":9,"SourceContext":"Seq.Server.Features.Indexing.EventStoreIndexer"}
{"@t":"2023-12-06T14:45:29.7820912Z","@mt":"Wrote {IndexSetCount} index sets in {Elapsed:0.000} ms","@r":["7.000"],"IndexSetCount":0,"Elapsed":7,"SourceContext":"Seq.Server.Features.Indexing.EventStoreIndexer"}
{"@t":"2023-12-06T14:45:31.9588488Z","@mt":"Applying {Count} retention policies","Count":1,"SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2023-12-06T14:45:31.9589466Z","@mt":"Removing all events before {DeletingUntilUtc:u} under policy {RetentionPolicyId}","@r":["2023-06-09 14:45:00Z"],"DeletingUntilUtc":"2023-06-09T14:45:00.0000000Z","RetentionPolicyId":"retentionpolicy-26","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2023-12-06T14:45:31.9684945Z","@mt":"Retention policy progressed in {Elapsed:0.000} ms (headway: {Headway}×), will resume from {ResumeStart:u}","@r":["9.518","2023-06-09 14:45:00Z"],"Elapsed":9.5179,"Headway":null,"ResumeStart":"2023-06-09T14:45:00.0000000Z","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor","RetentionPolicyId":"retentionpolicy-26"}
{"@t":"2023-12-06T14:45:31.9771887Z","@mt":"Retention processing took {Elapsed} ms; allocating {Allocation} ms for storage optimization","Elapsed":18.4681,"Allocation":599981.5319}
{"@t":"2023-12-06T14:45:42.5854220Z","@mt":"Coalesced {SpanCount} stable spans in {Elapsed:0.000} ms","@r":["10608.000"],"SpanCount":1,"Elapsed":10608,"SourceContext":"Seq.Server.Features.Retention.StorageOptimizer"}
{"@t":"2023-12-06T14:45:42.5855777Z","@mt":"Storage optimization took {Elapsed} ms; total iteration time was {Iteration} ms","Elapsed":10608,"Iteration":10626.7944}
{"@t":"2023-12-06T14:48:15.4933961Z","@mt":"Termination signal received"}
{"@t":"2023-12-06T14:48:15.8651914Z","@mt":"Signalling for {Workers} worker task(s) to complete","Workers":1,"SourceContext":"Seq.Engine.Workers.WorkerPool"}
{"@t":"2023-12-06T14:48:16.7540726Z","@mt":"Work {WorkDescription} ({WorkId}) observed cancellation and completed","WorkDescription":"Monitor alerts and trigger notifications","WorkId":"5f33aac3-ea20-4970-8998-d91018dbc628","SourceContext":"Seq.Engine.Workers.WorkerPool"}
{"@t":"2023-12-06T14:48:16.7729538Z","@mt":"Stopping {Tasks} scheduled tasks","Tasks":13,"SourceContext":"Seq.Server.Tasks.TaskRunner"}
{"@t":"2023-12-06T14:48:16.7832533Z","@mt":"Stopping input channel","SourceContext":"Seq.Server.Features.Apps.Inputs.InputChannel"}
{"@t":"2023-12-06T14:48:17.5363746Z","@mt":"Shutting down input channel; {AcceptedCount} queued events will be written","AcceptedCount":0,"SourceContext":"Seq.Server.Features.Apps.Inputs.InputChannel"}
{"@t":"2023-12-06T14:48:17.5419570Z","@mt":"Stopping broadcast channel","SourceContext":"Seq.Engine.Broadcast.BroadcastChannel"}
{"@t":"2023-12-06T14:48:17.9064950Z","@mt":"Shutting down broadcast channel with {AllowableFlush} flushable event batches (+{Tolerance} tolerance)","AllowableFlush":0,"Tolerance":15,"SourceContext":"Seq.Engine.Broadcast.BroadcastChannel"}
{"@t":"2023-12-06T14:48:18.3941276Z","@mt":"Stopping {AppCount} running apps","AppCount":1}
{"@t":"2023-12-06T14:48:19.8267797Z","@mt":"Closing event store","SourceContext":"Seq.Engine.Cluster.StorageNodeCoordinator"}
{"@t":"2023-12-06T14:48:19.8545767Z","@mt":"Closing metastore","SourceContext":"Quince.Storage.Native.NativeDocumentStore"}
{"@t":"2023-12-06T14:48:19.8942420Z","@mt":"Server stopped cleanly"}
bsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()","EventId":{"Id":6},"SourceContext":"Microsoft.AspNetCore.Hosting.Diagnostics"}
{"@t":"2023-12-06T14:47:57.3945679Z","@mt":"Error running the server application","@l":"Fatal","@x":"Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.\n ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.\n   caused by: snappy: error snappy reading buffered frame header\n   caused by: snappy: corrupt input (unsupported chunk length: 107)\n   at Flare.Ffi.Result.FlareResult.EnsureSuccess()\n   at Flare.Cursor.TryMoveNext()\n   at Quince.Storage.Native.NativeDocumentStore.PopulateRecordCache(Store store)\n   at Quince.Storage.Native.NativeDocumentStore..ctor(String path)\n   at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)\n   at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()\n   at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)\n   at Microsoft.AspNetCore.Hosting.WebHost.Start()\n   at Seq.ServiceProcess.ServerService.Start()"}
{"@t":"2023-12-06T14:47:57.4132410Z","@mt":"Failed to start server","@l":"Fatal","@x":"Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.\n ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.\n   caused by: snappy: error snappy reading buffered frame header\n   caused by: snappy: corrupt input (unsupported chunk length: 107)\n   at Flare.Ffi.Result.FlareResult.EnsureSuccess()\n   at Flare.Cursor.TryMoveNext()\n   at Quince.Storage.Native.NativeDocumentStore.PopulateRecordCache(Store store)\n   at Quince.Storage.Native.NativeDocumentStore..ctor(String path)\n   at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)\n   at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()\n   at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)\n   at Microsoft.AspNetCore.Hosting.WebHost.Start()\n   at Seq.ServiceProcess.ServerService.Start()\n   at Seq.Cli.RunCommand.RunStandardIOAsync(Func`1 bootstrap, TextWriter cout)"}
{"@t":"2023-12-06T14:48:12.5144796Z","@mt":"Seq {SeqVersion} running on OS {OSCaption}","SeqVersion":"2023.4.10219","OSCaption":"Linux 5.15.131.1-2.cm2 #1 SMP Sun Sep 24 03:38:45 UTC 2023","SourceContext":"Seq.Server.Features.Diagnostics.EnvironmentInformationWriter"}
{"@t":"2023-12-06T14:48:14.1573406Z","@mt":"Opening event store at {StreamPath}","StreamPath":"/data/Stream/stream.flare","SourceContext":"Seq.Engine.Cluster.StorageNodeCoordinator"}
{"@t":"2023-12-06T14:48:14.3408571Z","@mt":"Application startup exception","@l":"Fatal","@x":"Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.\n ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store.\n   caused by: failed to open a metadata store\n   caused by: failed to open the key-value store at: \"/data/Stream/stream.0.metadata\"\n   caused by: I/O error at path: \"/data/Stream/stream.0.metadata\"\n   caused by: No such file or directory (os error 2)\n   at Flare.Ffi.Result.FlareResult.EnsureSuccess()\n   at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)\n   at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)\n   at Flare.Store.OpenStandalone(String path, StoreLimits limits)\n   at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable`1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits)\n   at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits)\n   at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits)\n   at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters)\n   at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()","EventId":{"Id":6},"SourceContext":"Microsoft.AspNetCore.Hosting.Diagnostics"}
{"@t":"2023-12-06T14:48:14.3584887Z","@mt":"Error running the server application","@l":"Fatal","@x":"Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.\n ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store.\n   caused by: failed to open a metadata store\n   caused by: failed to open the key-value store at: \"/data/Stream/stream.0.metadata\"\n   caused by: I/O error at path: \"/data/Stream/stream.0.metadata\"\n   caused by: No such file or directory (os error 2)\n   at Flare.Ffi.Result.FlareResult.EnsureSuccess()\n   at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)\n   at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)\n   at Flare.Store.OpenStandalone(String path, StoreLimits limits)\n   at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable`1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits)\n   at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits)\n   at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits)\n   at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters)\n   at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()\n   at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)\n   at Microsoft.AspNetCore.Hosting.WebHost.Start()\n   at Seq.ServiceProcess.ServerService.Start()"}
{"@t":"2023-12-06T14:48:14.3692704Z","@mt":"Failed to start server","@l":"Fatal","@x":"Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.\n ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error attempting to open a metadata store.\n   caused by: failed to open a metadata store\n   caused by: failed to open the key-value store at: \"/data/Stream/stream.0.metadata\"\n   caused by: I/O error at path: \"/data/Stream/stream.0.metadata\"\n   caused by: No such file or directory (os error 2)\n   at Flare.Ffi.Result.FlareResult.EnsureSuccess()\n   at Flare.Ffi.Result.FlareResult.HandleNonSuccessWith(NonSuccessResultHandling handling)\n   at Flare.Ffi.Bindings.StoreOpen(IntPtr path, UIntPtr pathLen, IntPtr limits, StoreHandle& store, NonSuccessResultHandling nonSuccessResultHandling)\n   at Flare.Store.OpenStandalone(String path, StoreLimits limits)\n   at Seq.Engine.Storage.StorageEngine..ctor(Boolean standalone, String dataFile, PerformanceCounters sharedCounters, String nodeName, Nullable`1 initialRole, String listenUri, CertificateBundle listenTlsCertificate, String nextPeerUri, String authenticationKey, StorageEngineLimits limits)\n   at Seq.Engine.Storage.StorageEngine..ctor(String dataFile, PerformanceCounters sharedCounters, StorageEngineLimits limits)\n   at Seq.Engine.Cluster.StorageNodeCoordinator..ctor(String streamPath, StorageNodeConfiguration configuration, BroadcastChannel broadcast, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters, StorageNodeLimits limits)\n   at Seq.Cluster.ClusterNode..ctor(SeqConfig config, String nodeStateFilePath, String streamPath, WorkerPool workerPool, BroadcastChannel broadcast, IPortableEncryption encryption, IFilesystemCertificateStore certificateStore, PerformanceCounters sharedCounters)\n   at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)\n   at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)\n   at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)\n   at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext ctxt, Action`1 next)\n   at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   --- End of inner exception stack trace ---\n   at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)\n   at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)\n   at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)\n   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)\n   at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)\n   at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)\n   at Autofac.Core.Container.ResolveComponent(ResolveRequest request)\n   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)\n   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)\n   at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)\n   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)\n   at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)\n   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass5_0.<UseMiddleware>b__0(RequestDelegate next)\n   at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()\n   at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()\n   at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)\n   at Microsoft.AspNetCore.Hosting.WebHost.Start()\n   at Seq.ServiceProcess.ServerService.Start()\n   at Seq.Cli.RunCommand.RunStandardIOAsync(Func`1 bootstrap, TextWriter cout)"}
HakanL commented 11 months ago

I get this over and over:

2023-12-06T15:35:48.351999035Z [15:35:48 INF] Seq 2023.4.10219 running on OS Linux 5.15.131.1-2.cm2 #1 SMP Sun Sep 24 03:38:45 UTC 2023
2023-12-06T15:35:50.000149013Z [15:35:49 INF] Opening event store at /data/Stream/stream.flare
2023-12-06T15:35:50.290555012Z [15:35:50 INF] Ingestion enabled
2023-12-06T15:35:50.349534621Z [15:35:50 INF] Opening metastore /data/Documents/metastore.flare
2023-12-06T15:35:50.626346179Z [15:35:50 INF] Closing event store
2023-12-06T15:35:50.660314530Z Application startup exception: Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2023-12-06T15:35:50.660348230Z  ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.
2023-12-06T15:35:50.660373130Z    caused by: snappy: error snappy reading buffered frame header
2023-12-06T15:35:50.660378730Z    caused by: snappy: corrupt input (unsupported chunk length: 107)
2023-12-06T15:35:50.660401330Z    at Flare.Ffi.Result.FlareResult.EnsureSuccess()
HakanL commented 11 months ago

FWIW I'm using Azure Files, but I was under the impression that it's supported now without corruption, it's set to Premium LRS. This data in this SEQ instance isn't super-critical, and I'm happy to share files from it if it can help figuring out the issue. But I'd prefer not to wipe it if possible.

HakanL commented 11 months ago

Update: I was able to delete the files in the Documents folder and restore the last auto-backup, so I'm back up and running. But it would be interesting to know what could cause this, as I mentioned I thought using Azure Files would be supported? Or is there an issue in how I host it (App Service)?

KodrAus commented 11 months ago

Hi @HakanL 👋

Glad you got it all sorted. This looks like it was a data corruption. Unfortunately, finding properly reliable storage options in Azure is hard. Premium Files still occasionally corrupts, but I think it is as good as it gets outside of VMs or AKS. For more critical workloads, you can move your document storage to SQL Azure. We’ve got some docs on that here: https://docs.datalust.co/docs/using-sql-server-as-a-metastore

HakanL commented 11 months ago

@KodrAus Is the metastore more "sensitive" than the stream storage? Or are both susceptible to the corruption?

KodrAus commented 11 months ago

Both are susceptible to corruption but the stream storage is definitely more corruption resistant. Most of your event stream is chunked in immutable files, and if we encounter a corruption in recently ingested data then you might lose up to a few minutes worth of logs, which we just log as a metric typically.