search

datalust / seq-tickets

Issues, design discussions and feature roadmap for the Seq log server
https://datalust.co/seq
98 stars 5 forks source link

Seq crashes on start (previously worked) #2279

Closed apavelm closed 2 months ago

apavelm commented 2 months ago

I'm running seq inside docker, last night Seq unexpectedly stopped working. And refuse to restart. Exception stacktrace is below:

2024-09-23T05:31:38.741174298Z ────────────────────────────────────────
2024-09-23T05:31:38.741234800Z   Seq ♦ Machine data, for humans.
2024-09-23T05:31:38.789385382Z ─────────── © 2024 Datalust Pty Ltd ────
2024-09-23T05:31:38.789427783Z
2024-09-23T05:31:38.789446083Z Running as server; press Ctrl+C to exit.
2024-09-23T05:31:38.789458284Z
2024-09-23T05:31:39.731786762Z [05:31:39 INF] Seq 2024.3.12250 running on OS Ubuntu 22.04.4 LTS
2024-09-23T05:31:41.693084336Z [05:31:41 INF] Opening event store at /data/Stream/stream.flare
2024-09-23T05:31:42.053506135Z [05:31:42 INF] Ingestion enabled
2024-09-23T05:31:42.166861981Z [05:31:42 INF] Opening metastore /data/Documents/metastore.flare
2024-09-23T05:31:42.619646855Z [05:31:42 INF] Closing event store
2024-09-23T05:31:42.681115836Z Application startup exception: Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2024-09-23T05:31:42.681165737Z  ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.
2024-09-23T05:31:42.681175537Z    caused by: snappy: error snappy reading buffered frame header
2024-09-23T05:31:42.681182237Z    caused by: snappy: corrupt input (unsupported chunk length: 7368303)
2024-09-23T05:31:42.681188637Z    at Flare.Ffi.Result.FlareResult.EnsureSuccess()
2024-09-23T05:31:42.681195037Z    at Flare.Cursor.TryMoveNext()
2024-09-23T05:31:42.681201338Z    at Quince.Storage.Native.NativeDocumentStore.PopulateRecordCache(Store store)
2024-09-23T05:31:42.681207638Z    at Quince.Storage.Native.NativeDocumentStore..ctor(String path)
2024-09-23T05:31:42.681213938Z    at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)
2024-09-23T05:31:42.681220538Z    at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)
2024-09-23T05:31:42.681227138Z    at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)
2024-09-23T05:31:42.689853232Z    at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.689899633Z    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.690352943Z    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
2024-09-23T05:31:42.690412345Z    at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.690778753Z    at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.690804653Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709155166Z    at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709184166Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709191566Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709196567Z    --- End of inner exception stack trace ---
2024-09-23T05:31:42.709201867Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709206867Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709211767Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.709216267Z    at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709221067Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709226067Z    at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709230667Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709235467Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709240268Z    at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709245068Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709249868Z    at Autofac.Extensions.DependencyInjection.KeyedServiceMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.709254568Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.709259368Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.709265068Z    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request)
2024-09-23T05:31:42.709269868Z    at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request)
2024-09-23T05:31:42.709274468Z    at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest& request)
2024-09-23T05:31:42.709279168Z    at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.709292969Z    at Autofac.Core.Container.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.709297669Z    at Autofac.Core.Container.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.709302369Z    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
2024-09-23T05:31:42.709307169Z    at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
2024-09-23T05:31:42.709311769Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
2024-09-23T05:31:42.709316369Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
2024-09-23T05:31:42.709320969Z    at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
2024-09-23T05:31:42.709325569Z    at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
2024-09-23T05:31:42.709330170Z    at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
2024-09-23T05:31:42.709334970Z    at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.ReflectionMiddlewareBinder.CreateMiddleware(RequestDelegate next)
2024-09-23T05:31:42.709339670Z    at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
2024-09-23T05:31:42.709344270Z    at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
2024-09-23T05:31:42.747344724Z [05:31:42 FTL] Application startup exception
2024-09-23T05:31:42.747373524Z Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2024-09-23T05:31:42.747381325Z  ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.
2024-09-23T05:31:42.747386625Z    caused by: snappy: error snappy reading buffered frame header
2024-09-23T05:31:42.747391225Z    caused by: snappy: corrupt input (unsupported chunk length: 7368303)
2024-09-23T05:31:42.747395825Z    at Flare.Ffi.Result.FlareResult.EnsureSuccess()
2024-09-23T05:31:42.747400225Z    at Flare.Cursor.TryMoveNext()
2024-09-23T05:31:42.747404625Z    at Quince.Storage.Native.NativeDocumentStore.PopulateRecordCache(Store store)
2024-09-23T05:31:42.747409225Z    at Quince.Storage.Native.NativeDocumentStore..ctor(String path)
2024-09-23T05:31:42.747413725Z    at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)
2024-09-23T05:31:42.747418325Z    at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)
2024-09-23T05:31:42.747423025Z    at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)
2024-09-23T05:31:42.747436526Z    at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.747441726Z    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.747446726Z    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
2024-09-23T05:31:42.747451326Z    at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747456126Z    at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747460726Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.747465726Z    at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747470327Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.747476727Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747481427Z    --- End of inner exception stack trace ---
2024-09-23T05:31:42.747485927Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747490527Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.747495327Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.747499827Z    at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747504527Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.747509327Z    at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.747513928Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.747518728Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.747523528Z    at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.778780730Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.778789330Z    at Autofac.Extensions.DependencyInjection.KeyedServiceMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.778805831Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.778811031Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.778815731Z    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request)
2024-09-23T05:31:42.778820431Z    at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request)
2024-09-23T05:31:42.778825031Z    at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest& request)
2024-09-23T05:31:42.778834131Z    at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.778838931Z    at Autofac.Core.Container.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.778843631Z    at Autofac.Core.Container.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.778848232Z    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
2024-09-23T05:31:42.778853032Z    at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
2024-09-23T05:31:42.778857632Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
2024-09-23T05:31:42.778862132Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
2024-09-23T05:31:42.778866732Z    at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
2024-09-23T05:31:42.778871332Z    at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
2024-09-23T05:31:42.778875932Z    at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
2024-09-23T05:31:42.778880532Z    at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.ReflectionMiddlewareBinder.CreateMiddleware(RequestDelegate next)
2024-09-23T05:31:42.778885132Z    at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
2024-09-23T05:31:42.778889732Z    at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
2024-09-23T05:31:42.805637533Z [05:31:42 FTL] Error running the server application
2024-09-23T05:31:42.805670534Z Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2024-09-23T05:31:42.805678834Z  ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.
2024-09-23T05:31:42.805684134Z    caused by: snappy: error snappy reading buffered frame header
2024-09-23T05:31:42.805688735Z    caused by: snappy: corrupt input (unsupported chunk length: 7368303)
2024-09-23T05:31:42.805707435Z    at Flare.Ffi.Result.FlareResult.EnsureSuccess()
2024-09-23T05:31:42.805712235Z    at Flare.Cursor.TryMoveNext()
2024-09-23T05:31:42.805716835Z    at Quince.Storage.Native.NativeDocumentStore.PopulateRecordCache(Store store)
2024-09-23T05:31:42.805721435Z    at Quince.Storage.Native.NativeDocumentStore..ctor(String path)
2024-09-23T05:31:42.805726035Z    at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)
2024-09-23T05:31:42.805730736Z    at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)
2024-09-23T05:31:42.805735536Z    at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)
2024-09-23T05:31:42.805740236Z    at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.805745336Z    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.805750236Z    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
2024-09-23T05:31:42.805754936Z    at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805759636Z    at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805764336Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805769336Z    at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805774036Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805779937Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805784737Z    --- End of inner exception stack trace ---
2024-09-23T05:31:42.805789237Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805793937Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805798837Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.805803437Z    at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805808137Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805813037Z    at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805822538Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805827438Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805832238Z    at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805836838Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805841538Z    at Autofac.Extensions.DependencyInjection.KeyedServiceMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.805846138Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.805850938Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.805855438Z    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request)
2024-09-23T05:31:42.805860138Z    at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request)
2024-09-23T05:31:42.805864839Z    at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest& request)
2024-09-23T05:31:42.805869639Z    at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.805874339Z    at Autofac.Core.Container.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.805878839Z    at Autofac.Core.Container.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.805883439Z    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
2024-09-23T05:31:42.805888139Z    at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
2024-09-23T05:31:42.805892839Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
2024-09-23T05:31:42.805897339Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
2024-09-23T05:31:42.805901939Z    at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
2024-09-23T05:31:42.805906439Z    at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
2024-09-23T05:31:42.805911040Z    at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
2024-09-23T05:31:42.805915740Z    at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.ReflectionMiddlewareBinder.CreateMiddleware(RequestDelegate next)
2024-09-23T05:31:42.805920340Z    at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
2024-09-23T05:31:42.805928840Z    at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
2024-09-23T05:31:42.805933640Z    at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)
2024-09-23T05:31:42.805938140Z    at Microsoft.AspNetCore.Hosting.WebHost.Start()
2024-09-23T05:31:42.805942540Z    at Seq.ServiceProcess.ServerService.Start()
2024-09-23T05:31:42.847579576Z [05:31:42 FTL] Failed to start server
2024-09-23T05:31:42.847611376Z Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem.
2024-09-23T05:31:42.847619877Z  ---> Flare.Ffi.Result.FlareException: Flare native storage failed (IOError), error evaluating the next document to read.
2024-09-23T05:31:42.847625377Z    caused by: snappy: error snappy reading buffered frame header
2024-09-23T05:31:42.847630677Z    caused by: snappy: corrupt input (unsupported chunk length: 7368303)
2024-09-23T05:31:42.847635777Z    at Flare.Ffi.Result.FlareResult.EnsureSuccess()
2024-09-23T05:31:42.847640977Z    at Flare.Cursor.TryMoveNext()
2024-09-23T05:31:42.847645777Z    at Quince.Storage.Native.NativeDocumentStore.PopulateRecordCache(Store store)
2024-09-23T05:31:42.847651177Z    at Quince.Storage.Native.NativeDocumentStore..ctor(String path)
2024-09-23T05:31:42.847656277Z    at Seq.Server.StorageSubsystem.ConstructInnerMetastore(String documentsPath, SeqConfig config)
2024-09-23T05:31:42.847661278Z    at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters)
2024-09-23T05:31:42.847666778Z    at Seq.Server.SeqServerModule.<Load>b__9_10(IComponentContext c)
2024-09-23T05:31:42.847671878Z    at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.<Register>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.847677278Z    at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
2024-09-23T05:31:42.847682578Z    at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
2024-09-23T05:31:42.847687678Z    at Autofac.Core.Activators.Delegate.DelegateActivator.<ConfigurePipeline>b__2_0(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847692878Z    at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847697878Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847703279Z    at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847708279Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847727179Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847732579Z    --- End of inner exception stack trace ---
2024-09-23T05:31:42.847737479Z    at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847742479Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847747680Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.847752680Z    at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847757780Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847763080Z    at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847768080Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847773480Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847778680Z    at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847783780Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847788880Z    at Autofac.Extensions.DependencyInjection.KeyedServiceMiddleware.Execute(ResolveRequestContext context, Action`1 next)
2024-09-23T05:31:42.847793881Z    at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context)
2024-09-23T05:31:42.847799081Z    at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context)
2024-09-23T05:31:42.847804181Z    at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request)
2024-09-23T05:31:42.847809281Z    at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request)
2024-09-23T05:31:42.847814281Z    at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest& request)
2024-09-23T05:31:42.847819581Z    at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.847824681Z    at Autofac.Core.Container.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.847829681Z    at Autofac.Core.Container.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request)
2024-09-23T05:31:42.847834682Z    at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
2024-09-23T05:31:42.847844482Z    at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
2024-09-23T05:31:42.847849882Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
2024-09-23T05:31:42.847854882Z    at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType)
2024-09-23T05:31:42.847859882Z    at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
2024-09-23T05:31:42.847864782Z    at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
2024-09-23T05:31:42.847869782Z    at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
2024-09-23T05:31:42.847874882Z    at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.ReflectionMiddlewareBinder.CreateMiddleware(RequestDelegate next)
2024-09-23T05:31:42.847879883Z    at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build()
2024-09-23T05:31:42.847884783Z    at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()
2024-09-23T05:31:42.847889683Z    at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(CancellationToken cancellationToken)
2024-09-23T05:31:42.847894583Z    at Microsoft.AspNetCore.Hosting.WebHost.Start()
2024-09-23T05:31:42.847899483Z    at Seq.ServiceProcess.ServerService.Start()
2024-09-23T05:31:42.847904383Z    at Seq.Cli.RunCommand.RunStandardIOAsync(Func`1 bootstrap, TextWriter cout)
KodrAus commented 2 months ago

Hi @apavelm :wave:

It looks like you've got a disk corruption that's preventing Seq from starting. You should be able to recover your instance by following the backup and restore process.

If you're running in Azure Container Instances, we recommend using SQL Azure as your document store because it's more reliable than Azure Files. If you use a SQL Azure document store, your log data will still be stored on Azure Files, but the scope of corruptions there are smaller. Those docs for SQL Azure are based on Windows, but in Docker the instructions are the same except that the seq command is called seqsvr.

apavelm commented 2 months ago

Hi @KodrAus ! Thank you for the link. Indeed, I'm using Azure Files. I'm trying to restore, but no luck so far. Anywaym thank you!

KodrAus commented 2 months ago

Hmm, what if you move/rename your entire C:\ProgramData\Seq\Documents folder before attempting to restore the backup? If you rename it to C:\ProgramData\Seq\DocumentsBackup then the restore should try work off a fully clean state.

apavelm commented 2 months ago

Hmm, what if you move/rename your entire C:\ProgramData\Seq\Documents folder before attempting to restore the backup? If you rename it to C:\ProgramData\Seq\DocumentsBackup then the restore should try work off a fully clean state.

I tried this option as well. Didn't help. So, I've just dropped everything and reconfigured all parameters manually from the scratch