search

OctopusDeploy / Issues

| Public | Bug reports and known issues for Octopus Deploy and all related tools
https://octopus.com
162 stars 20 forks source link

Account usage in large variable sets can cause excessive memory usage #8854

Closed benPearce1 closed 3 months ago

benPearce1 commented 3 months ago

Severity

Blocking some customers, no workaround available

Version

Tested on 2024.3

Latest Version

I could reproduce the problem in the latest build

What happened?

For an account which is referenced in a variable with a large number of variable snapshots (for example, modified variable sets that are linked to a large number of releases), viewing the usages can result in excessive memory usage on the server, possibly resulting in an Out of Memory exception.

Reproduction

  1. Create an account
  2. Create a project
  3. Create a variable to that account
  4. Create a large number of other variables, particularly with very long values.
  5. Create a release
  6. Modify any variable or add a variable on the project
  7. Repeat steps 4 and 5 hundreds or thousands of times.
  8. Go to account -> usage.
  9. The result will be slow or cause an error.

Error and Stacktrace

---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)
   at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
   at System.Text.StringBuilder.AppendWithExpansion(Char& value, Int32 valueCount)
   at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount)
   at System.IO.StringWriter.Write(Char[] buffer, Int32 index, Int32 count)
   at Newtonsoft.Json.Utilities.JavaScriptUtils.WriteEscapedJavaScriptString(TextWriter writer, String s, Char delimiter, Boolean appendDelimiters, Boolean[] charEscapeFlags, StringEscapeHandling stringEscapeHandling, IArrayPool`1 bufferPool, Char[]& writeBuffer)
   at Newtonsoft.Json.JsonTextWriter.WriteValue(String value)
   at Newtonsoft.Json.JsonWriter.WriteToken(JsonReader reader, Boolean writeChildren, Boolean writeDateConstructorAsDate, Boolean writeComments)
   at Newtonsoft.Json.Serialization.TraceJsonReader.Read()
   at Newtonsoft.Json.Linq.JContainer.ReadContentFrom(JsonReader r, JsonLoadSettings settings)
   at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
   at Octopus.Core.RelationalStorage.VariableConverter.ReadJsonInternal(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in ./source/Octopus.Core/RelationalStorage/VariableConverter.cs:line 68
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Nevermore.Advanced.Serialization.NewtonsoftDocumentSerializer.DeserializeLargeText(TextReader reader, Type type)
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderContext.DeserializeText[TDocument](DbDataReader reader, Int32 index, Type concreteType)
   at lambda_method10052(Closure, DbDataReader, DocumentReaderContext)
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderStrategy.<>c__DisplayClass4_1`1.<CreateReader>b__1(DbDataReader dbDataReader)
   --- End of inner exception stack trace ---
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderStrategy.<>c__DisplayClass4_1`1.<CreateReader>b__1(DbDataReader dbDataReader)
   at Nevermore.Advanced.ReadTransaction.ProcessReader[TRecord](DbDataReader reader, PreparedCommand command)+MoveNext()
   at Nevermore.Advanced.ReadTransaction.<>c__DisplayClass112_0`1.<<Stream>g__Execute|0>d.MoveNext()
   at Octopus.Core.Persistence.Database.Indexes.DeletionDocumentUsageFinder.RelationshipBuilderNevermore`3.FindUsages(IReadQueryExecutor queryExecutor, Object from, List`1 referencingDocuments) in ./source/Octopus.Core/Persistence/Database/Indexes/DeletionDocumentUsageFinder.cs:line 210
   at Octopus.Core.Persistence.Database.Indexes.DeletionDocumentUsageFinder.FindReferences[TDocument](IOctopusQueryExecutor queryExecutor, TDocument document)
   at Octopus.Core.Features.Accounts.AccountUsageService.BuildAccountUsageResource(Account account) in ./source/Octopus.Core/Features/Accounts/AccountUsageService.cs:line 51
   at Octopus.Core.Features.Accounts.GetAccountUsageRequestHandler.Handle(GetAccountUsageRequest request, CancellationToken cancellationToken) in ./source/Octopus.Core/Features/Accounts/GetAccountUsageRequestHandler.cs:line 24
   at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken)
   at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken)
   at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken)
   at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Request[TRequest,TResponse](IRequest`2 request, CancellationToken cancellationToken)
   at Octopus.Server.Web.Controllers.Accounts.GetAccountUsageController.GetAccountUsage(GetAccountUsageRequest request, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Controllers/Accounts/GetAccountUsageController.cs:line 20
   at lambda_method160913(Closure, Object)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Octopus.Server.Web.Middleware.BoundaryTrailerRewriteMiddleware.Invoke(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/BoundaryTrailerRewriteMiddleware.cs:line 45
   at Octopus.Server.Web.Infrastructure.Authentication.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) in ./source/Octopus.Server/Web/Infrastructure/Authentication/AuthorizationMiddlewareResultHandler.cs:line 50
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 31
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 24
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 50
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 58
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, IUserAuthenticator userAuthenticator, IUserSessionService userSessionService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 61
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.LegacyRequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/LegacyRequestLoggerMiddleware.cs:line 35
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 64
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 51

More Information

No response

Workaround

Tighten up release retention to remove old variable sets in the database.

octoreleasebot commented 3 months ago

Release Note: Improved performance of account usage across variable snapshots

octoreleasebot commented 3 months ago

Release Note: Improved performance of account usage across variable snapshots

Octobob commented 4 weeks ago

:tada: The fix for this issue has been released in:

Release stream Release
2024.3 2024.3.2808
2024.4+ all releases