search

datalust / seq-tickets

Issues, design discussions and feature roadmap for the Seq log server
https://datalust.co/seq
97 stars 5 forks source link

Load Issue on Seq 2024.3.12250 #2291

Closed Jaben closed 2 weeks ago

Jaben commented 4 weeks ago

Describe the bug On Startup:

[12:57:29 INF] Seq 2024.3.12250 running on OS Ubuntu 22.04.4 LTS [12:57:29 INF] Opening event store at /data/Stream/stream.flare [12:57:29 INF] Ingestion enabled [12:57:29 INF] Opening metastore /data/Documents/metastore.flare [12:57:29 INF] Applying migration Disable legacy event schema compatibility if possible [12:57:29 INF] Closing metastore [12:57:29 INF] Closing event store Application startup exception: Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Seq.Cluster.IClusterNode -> λ:Seq.Server.StorageSubsystem. ---> System.ArgumentNullException: Value cannot be null. (Parameter 's') at System.DateTime.Parse(String s) at Seq.Data.Storage.Migrations.DisableLegacyEventSchemaCompatibilityIfPossible.Apply(DocumentSession session) at Quince.Storage.Native.NativeDocumentStore.Migrate(MigrationList migrations) at Seq.Server.StorageSubsystem.Initialize(IComponentContext ctx, SeqConfig config, StorageLayout layout, PerformanceCounters sharedCounters) at Seq.Server.SeqServerModule.b9_10(IComponentContext c) at Autofac.RegistrationExtensions.<>c__DisplayClass41_0`1.b0(IComponentContext c, IEnumerable1 p) at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_01.b0(IComponentContext c, IEnumerable1 p) at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable1 parameters) at Autofac.Core.Activators.Delegate.DelegateActivator.b2_0(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>cDisplayClass14_0.b1(ResolveRequestContext context) at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>cDisplayClass14_0.b1(ResolveRequestContext context) at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) --- End of inner exception stack trace --- at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>cDisplayClass14_0.b1(ResolveRequestContext context) at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context) at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>cDisplayClass14_0.b1(ResolveRequestContext context) at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext context) at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>cDisplayClass14_0.b1(ResolveRequestContext context) at Autofac.Extensions.DependencyInjection.KeyedServiceMiddleware.Execute(ResolveRequestContext context, Action`1 next) at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>cDisplayClass14_0.b1(ResolveRequestContext context) at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext context) at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest& request) at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest& request) at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest& request) at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest& request) at Autofac.Core.Container.ResolveComponent(ResolveRequest& request) at Autofac.Core.Container.Autofac.IComponentContext.ResolveComponent(ResolveRequest& request) at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType, IEnumerable`1 parameters) at Autofac.ResolutionExtensions.ResolveOptional(IComponentContext context, Type serviceType) at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) at Microsoft.Extensions.Internal.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider) at Microsoft.Extensions.Internal.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters) at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.ReflectionMiddlewareBinder.CreateMiddleware(RequestDelegate next) at Microsoft.AspNetCore.Builder.ApplicationBuilder.Build() at Microsoft.AspNetCore.Hosting.WebHost.BuildApplication()

To Reproduce Startup Seq 2024.3.12250

Expected behavior Not fail on startup.

Screenshots image

Environment (please complete the following information): Docker running in OS Ubuntu 22.04.4 LTS -- Host is Ubuntu 20.

Jaben commented 4 weeks ago

My solution for now is to keep running 2024.2 -- which still works.

nblumhardt commented 3 weeks ago

Thanks for the report, @Jaben. We've pulled together a fix that should address this, although we're not sure of the sequence of events — backups, restores, upgrades, etc. — that put the configuration in the problematic state. The issue could have been in any version since late 5.1.x so it might have to remain a mystery (if this is a new instance please let us know, though).

I'm not sure of an ETA yet but expect it'll be within the next few weeks.

Jaben commented 3 weeks ago

I can zip the metadata and email it to you. What files should I include?

nblumhardt commented 3 weeks ago

Thanks for the offer! It should be possible to extract the value in question without sending any files, I'll just need a chance to craft a command line to extract it, which I'll take a shot at on Monday 👍

Jaben commented 2 weeks ago

I have another production SEQ instance failed to start because of this issue. Any update on a fix?

nblumhardt commented 2 weeks ago

Thanks for the update; we'll push this forward and aim to have a build out this week.

nblumhardt commented 2 weeks ago

(Closing as the fix is committed to the release branch now)

nblumhardt commented 1 week ago

2024.3.12680 should have this one resolved; if you still hit any problems please let us know. Thanks again for the report!

Jaben commented 1 week ago

Thanks, Nick!

On Thu, Oct 24, 2024, at 8:51 PM, Nicholas Blumhardt wrote:

2024.3.12680 should have this one resolved; if you still hit any problems please let us know. Thanks again for the report!

— Reply to this email directly, view it on GitHub https://github.com/datalust/seq-tickets/issues/2291#issuecomment-2436588131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADNRVVCZX47LYR72BGQHELZ5GIYJAVCNFSM6AAAAABPWXSGU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWGU4DQMJTGE. You are receiving this because you were mentioned.Message ID: @.***>