`Syntax error: maximum expression complexity reached` with multiple signals in group selected

[MaceWindu]

Describe the bug

I've created a group for signals to be able to work with UNION of signals and it didn't worked out well as it gives following error when all signals in group selected:

Flare.Ffi.Result.SyntaxErrorException: Syntax error: maximum expression complexity reached.
   at Seq.Engine.Storage.StorageEngine.Query(Span`1 query, Object[] sharedColumnBuffer, CancellationToken cancel)
   at Seq.Engine.Events.EventStore.Query(Span`1 query, Object[] sharedColumnBuffer, Boolean disableReadRateLimit, CancellationToken cancel)
   at Seq.Engine.Queries.DataStore.Get(EventStoreRange range, RetrievalStatistics statistics, CancellationToken cancel)+MoveNext()
   at System.Linq.Enumerable.EnumerablePartition`1.ToArray()
   at Seq.Engine.Queries.DataStore.Search(Int32 count, DateTimeRangeBounds bounds, String filter, IndexExpression indexExpression, Nullable`1 startFrom, Nullable`1 startAfter, Nullable`1 shortCircuitAfter, TraversalDirection direction, Boolean background, CancellationToken& cancel)
   at Seq.Server.Web.Api.EventsController.<>c__DisplayClass20_0.<SearchEvents>b__1(CancellationToken ct)
   at Seq.Engine.Workers.WorkerPool.<>c__DisplayClass7_0`1.<Run>g__DoWork|0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Seq.Server.Web.Api.EventsController.SearchEvents(IndexedFilter indexedFilter)
   at Seq.Server.Web.Api.EventsController.InSignal(EvaluationContext evaluationContext, Boolean excludeStats)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.RateLimiting.RateLimitingMiddleware.InvokeInternal(HttpContext context, EnableRateLimitingAttribute enableRateLimitingAttribute)
   at Seq.Server.Web.Middleware.WebSocketAcceptMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.RequestAuthenticationMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.ServerStatusMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.UniversalHeadersMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.RequestCompletionMiddleware.Invoke(HttpContext httpContext)

Deselecting several signals helps, but defeat purpose of using group.

To Reproduce

I suspect it is easy to reproduce with several signals with big set of filters, but if needed I can provide some. I suspect all signal filters ORed into single filter, which doesn't work well with filter expression size limits.

Expected behavior

As group query size expected to be a sum of sizes of contained signals, probably it doesn't make sense to enforce limits here?

Environment (please complete the following information):

• OS: Windows
• Seq Version: 2024.3.12021
• Using Docker? No

[MaceWindu]

Currently resolved configuring options from here https://github.com/datalust/seq-tickets/issues/2086#issuecomment-2005697748 to bigger numbers

[nblumhardt]

Thanks for the heads-up. This one is by design, the expression complexity limit is a safety mechanism to avoid instability when processing very deeply nested expressions (essentially a "soft" stack overflow exception).

Increasing the limit is not harmful but may increase the likelihood of crashes. Is there any way to "garbage collect" these very complex filters to produce simpler ones, e.g. by removing duplicated or redundant clauses?

[MaceWindu]

We cleanup signals from time to time from obsoleted filters, but I don't think it will provide any noticeable reduction in size of filter.

In any case, I think I'm fine with limits change workaround, so I will close it.